From 9e82551704e1020773b772689e1b6b099cc2e75a Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Tue, 12 May 2026 17:25:13 -0600 Subject: [PATCH 01/14] feat: initial seed of Apache DataFusion Java bindings Seed the project with a minimal end-to-end JNI binding from the JVM to Apache DataFusion, plus the build, format, and license-check tooling needed for ongoing contribution. Java surface (org.apache.datafusion): - SessionContext: AutoCloseable session, sql(String) returning a lazy DataFrame, registerParquet(String, String) for registering local Parquet files as SQL tables. - DataFrame: AutoCloseable, collect(BufferAllocator) executes the plan and returns result batches as an Arrow ArrowReader via the Arrow C Data Interface. collect() consumes the DataFrame; close() releases the native plan if never collected. Native side (native/, crate datafusion-jni): - JNI entry points for SessionContext create/close/registerParquet/ createDataFrame and DataFrame collect/close. - Results are exported as FFI_ArrowArrayStream so the JVM reads batches without per-row JNI crossings or row-by-row copies. Build and contributor tooling: - pom.xml with Maven wrapper, JUnit 5, Arrow 19, JDK 17 toolchain. - apache-rat-plugin (license-header check) and spotless-maven-plugin (google-java-format) both bound to the verify phase. - Makefile targets for native build, JVM build, test, clean, and TPC-H SF1 test data generation via tpchgen-cli. - GitHub Actions workflow running spotless:check and cargo fmt --check on push and pull_request to main. --- .github/workflows/format.yml | 50 + .gitignore | 9 + .mvn/wrapper/maven-wrapper.properties | 3 + LICENSE.txt | 213 ++ Makefile | 39 + NOTICE.txt | 5 + README.md | 129 +- mvnw | 295 ++ mvnw.cmd | 189 + native/Cargo.lock | 3370 +++++++++++++++++ native/Cargo.toml | 31 + native/src/errors.rs | 58 + native/src/lib.rs | 154 + pom.xml | 143 + .../java/org/apache/datafusion/DataFrame.java | 84 + .../datafusion/NativeLibraryLoader.java | 33 + .../org/apache/datafusion/SessionContext.java | 78 + .../apache/datafusion/SessionContextTest.java | 93 + 18 files changed, 4975 insertions(+), 1 deletion(-) create mode 100644 .github/workflows/format.yml create mode 100644 .gitignore create mode 100644 .mvn/wrapper/maven-wrapper.properties create mode 100644 LICENSE.txt create mode 100644 Makefile create mode 100644 NOTICE.txt create mode 100755 mvnw create mode 100644 mvnw.cmd create mode 100644 native/Cargo.lock create mode 100644 native/Cargo.toml create mode 100644 native/src/errors.rs create mode 100644 native/src/lib.rs create mode 100644 pom.xml create mode 100644 src/main/java/org/apache/datafusion/DataFrame.java create mode 100644 src/main/java/org/apache/datafusion/NativeLibraryLoader.java create mode 100644 src/main/java/org/apache/datafusion/SessionContext.java create mode 100644 src/test/java/org/apache/datafusion/SessionContextTest.java diff --git a/.github/workflows/format.yml b/.github/workflows/format.yml new file mode 100644 index 0000000..6295c32 --- /dev/null +++ b/.github/workflows/format.yml @@ -0,0 +1,50 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +name: Format + +on: + push: + branches: [main] + pull_request: + branches: [main] + +jobs: + java: + name: Java (spotless) + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-java@v4 + with: + distribution: temurin + java-version: '17' + cache: maven + - name: Check Java formatting + run: ./mvnw -B -ntp spotless:check + + rust: + name: Rust (cargo fmt) + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: dtolnay/rust-toolchain@stable + with: + components: rustfmt + - name: Check Rust formatting + working-directory: native + run: cargo fmt --all -- --check diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..f19fb42 --- /dev/null +++ b/.gitignore @@ -0,0 +1,9 @@ +target/ +*.class +.idea/ +.vscode/ +*.iml +.DS_Store +tpch-data/ +.claude +docs/superpowers diff --git a/.mvn/wrapper/maven-wrapper.properties b/.mvn/wrapper/maven-wrapper.properties new file mode 100644 index 0000000..ffcab66 --- /dev/null +++ b/.mvn/wrapper/maven-wrapper.properties @@ -0,0 +1,3 @@ +wrapperVersion=3.3.4 +distributionType=only-script +distributionUrl=https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.9.9/apache-maven-3.9.9-bin.zip diff --git a/LICENSE.txt b/LICENSE.txt new file mode 100644 index 0000000..920daaf --- /dev/null +++ b/LICENSE.txt @@ -0,0 +1,213 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +-------------------------------------------------------------------------------- + +This project includes code from Apache Aurora. + +* dev/release/{release,changelog,release-candidate} are based on the scripts from + Apache Aurora + +Copyright: 2016 The Apache Software Foundation. +Home page: https://aurora.apache.org/ +License: http://www.apache.org/licenses/LICENSE-2.0 \ No newline at end of file diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..ca894d3 --- /dev/null +++ b/Makefile @@ -0,0 +1,39 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +.PHONY: all native jvm test clean tpch-data + +all: native jvm + +native: + cd native && cargo build + +jvm: + ./mvnw package -DskipTests + +test: native + ./mvnw test + +clean: + cd native && cargo clean + ./mvnw clean + +tpch-data: + @command -v tpchgen-cli >/dev/null || \ + (echo "Install: cargo install tpchgen-cli" && exit 1) + mkdir -p tpch-data/sf1 + tpchgen-cli -s 1 -f parquet -o tpch-data/sf1 diff --git a/NOTICE.txt b/NOTICE.txt new file mode 100644 index 0000000..d46ff73 --- /dev/null +++ b/NOTICE.txt @@ -0,0 +1,5 @@ +Apache DataFusion Java +Copyright 2026 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). diff --git a/README.md b/README.md index 9635952..f613008 100644 --- a/README.md +++ b/README.md @@ -1 +1,128 @@ -# DataFusion Java Bindings +# Apache DataFusion Java + +Java bindings for [Apache DataFusion], the Rust-based query engine. SQL queries +run in native code and results are returned to the JVM as [Apache Arrow] record +batches via the Arrow C Data Interface — no per-row JNI calls, no row-by-row +copies. + +[Apache DataFusion]: https://datafusion.apache.org/ +[Apache Arrow]: https://arrow.apache.org/ + +> **Project status: early development.** This is a brand-new project. The API +> is small, will change without notice, and there is no published release. Do +> not depend on it from production code yet. Bug reports, design feedback, and +> contributions are very welcome. + +## Quickstart + +```java +import org.apache.arrow.memory.RootAllocator; +import org.apache.arrow.vector.ipc.ArrowReader; +import org.apache.datafusion.DataFrame; +import org.apache.datafusion.SessionContext; + +try (var allocator = new RootAllocator(); + var ctx = new SessionContext()) { + + ctx.registerParquet("orders", "/path/to/orders.parquet"); + + try (DataFrame df = ctx.sql( + "SELECT o_orderpriority, COUNT(*) AS n " + + "FROM orders GROUP BY o_orderpriority"); + ArrowReader reader = df.collect(allocator)) { + while (reader.loadNextBatch()) { + var batch = reader.getVectorSchemaRoot(); + // ... consume batch ... + } + } +} +``` + +The current public surface mirrors a small slice of the Rust DataFusion API: + +- `SessionContext.sql(String query)` — parse and plan a SQL query, returning a + lazy `DataFrame`. No execution happens yet. +- `DataFrame.collect(BufferAllocator allocator)` — execute the plan and return + the result batches as an `ArrowReader`. Consumes the DataFrame; the caller + closes the reader, and the allocator must outlive it. +- `SessionContext.registerParquet(String name, String path)` — register a local + Parquet file as a SQL table. + +Both `SessionContext` and `DataFrame` are `AutoCloseable` and **not +thread-safe**. + +## Prerequisites + +- JDK 17 or newer +- Rust toolchain (stable, installed via [rustup]) +- [`tpchgen-cli`] — only needed to generate test data for the Parquet + integration test (`cargo install tpchgen-cli`) + +Maven is bundled via the `./mvnw` wrapper; no separate Maven install required. + +[rustup]: https://rustup.rs/ +[`tpchgen-cli`]: https://github.com/clflushopt/tpchgen-rs + +## Build & test + + make test + +This builds the native Rust crate and runs the JUnit tests. The steps can be +run individually: + + cd native && cargo build + ./mvnw test + +The native library must be built before running JVM tests. + +## Test data + +The Parquet integration test reads TPC-H SF1 data (~345 MB across 8 tables in +Snappy-compressed Parquet). Generate it once with: + + make tpch-data + +Tests that need this data skip cleanly if it is missing. `make clean` does +**not** remove `tpch-data/` — delete it manually to reclaim the disk space. + +## Repository layout + +- `src/` — Java sources and tests +- `native/` — Rust crate that exposes DataFusion over JNI and the Arrow C Data + Interface + +## Roadmap + +Near-term priorities, roughly in order: + +- **Session configuration.** Expose `SessionConfig` and `RuntimeEnv` settings + (target partitions, batch size, memory pool, default catalog, …) so callers + can tune execution from the JVM. +- **Full `SessionContext` and `DataFrame` APIs.** Expand beyond `sql` and + `registerParquet` to mirror the Rust API: table registration variants, + `read_*` / `write_*` entry points, and DataFrame transformations such as + `select`, `filter`, `join`, `aggregate`, `sort`, plus result-materialization + variants (`show`, `count`, streaming collection). +- **JVM-side plan construction via Protobuf.** Build DataFusion logical and + physical plans on the JVM using the existing DataFusion Protobuf + representation, then ship them to the native side for execution. This lets + plans assembled by other JVM tools (Spark, Flink, Beam, custom planners) run + on DataFusion without having to go through SQL. +- **Java-defined expressions.** Allow user expressions / UDFs to be implemented + in Java and invoked from the native plan, operating on Apache Arrow vectors + (Java side) so evaluation stays vectorized end-to-end and avoids row-by-row + JNI crossings. + +These are intentionally large items — design discussion via GitHub issues +before implementation is welcome. + +## Contributing + +This project follows the Apache DataFusion contribution model. Issues and pull +requests are welcome — please open a GitHub issue to discuss any significant +change before sending a PR. + +## License + +Licensed under the [Apache License, Version 2.0](LICENSE.txt). See +[NOTICE.txt](NOTICE.txt) for required attributions. diff --git a/mvnw b/mvnw new file mode 100755 index 0000000..bd8896b --- /dev/null +++ b/mvnw @@ -0,0 +1,295 @@ +#!/bin/sh +# ---------------------------------------------------------------------------- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# ---------------------------------------------------------------------------- + +# ---------------------------------------------------------------------------- +# Apache Maven Wrapper startup batch script, version 3.3.4 +# +# Optional ENV vars +# ----------------- +# JAVA_HOME - location of a JDK home dir, required when download maven via java source +# MVNW_REPOURL - repo url base for downloading maven distribution +# MVNW_USERNAME/MVNW_PASSWORD - user and password for downloading maven +# MVNW_VERBOSE - true: enable verbose log; debug: trace the mvnw script; others: silence the output +# ---------------------------------------------------------------------------- + +set -euf +[ "${MVNW_VERBOSE-}" != debug ] || set -x + +# OS specific support. +native_path() { printf %s\\n "$1"; } +case "$(uname)" in +CYGWIN* | MINGW*) + [ -z "${JAVA_HOME-}" ] || JAVA_HOME="$(cygpath --unix "$JAVA_HOME")" + native_path() { cygpath --path --windows "$1"; } + ;; +esac + +# set JAVACMD and JAVACCMD +set_java_home() { + # For Cygwin and MinGW, ensure paths are in Unix format before anything is touched + if [ -n "${JAVA_HOME-}" ]; then + if [ -x "$JAVA_HOME/jre/sh/java" ]; then + # IBM's JDK on AIX uses strange locations for the executables + JAVACMD="$JAVA_HOME/jre/sh/java" + JAVACCMD="$JAVA_HOME/jre/sh/javac" + else + JAVACMD="$JAVA_HOME/bin/java" + JAVACCMD="$JAVA_HOME/bin/javac" + + if [ ! -x "$JAVACMD" ] || [ ! -x "$JAVACCMD" ]; then + echo "The JAVA_HOME environment variable is not defined correctly, so mvnw cannot run." >&2 + echo "JAVA_HOME is set to \"$JAVA_HOME\", but \"\$JAVA_HOME/bin/java\" or \"\$JAVA_HOME/bin/javac\" does not exist." >&2 + return 1 + fi + fi + else + JAVACMD="$( + 'set' +e + 'unset' -f command 2>/dev/null + 'command' -v java + )" || : + JAVACCMD="$( + 'set' +e + 'unset' -f command 2>/dev/null + 'command' -v javac + )" || : + + if [ ! -x "${JAVACMD-}" ] || [ ! -x "${JAVACCMD-}" ]; then + echo "The java/javac command does not exist in PATH nor is JAVA_HOME set, so mvnw cannot run." >&2 + return 1 + fi + fi +} + +# hash string like Java String::hashCode +hash_string() { + str="${1:-}" h=0 + while [ -n "$str" ]; do + char="${str%"${str#?}"}" + h=$(((h * 31 + $(LC_CTYPE=C printf %d "'$char")) % 4294967296)) + str="${str#?}" + done + printf %x\\n $h +} + +verbose() { :; } +[ "${MVNW_VERBOSE-}" != true ] || verbose() { printf %s\\n "${1-}"; } + +die() { + printf %s\\n "$1" >&2 + exit 1 +} + +trim() { + # MWRAPPER-139: + # Trims trailing and leading whitespace, carriage returns, tabs, and linefeeds. + # Needed for removing poorly interpreted newline sequences when running in more + # exotic environments such as mingw bash on Windows. + printf "%s" "${1}" | tr -d '[:space:]' +} + +scriptDir="$(dirname "$0")" +scriptName="$(basename "$0")" + +# parse distributionUrl and optional distributionSha256Sum, requires .mvn/wrapper/maven-wrapper.properties +while IFS="=" read -r key value; do + case "${key-}" in + distributionUrl) distributionUrl=$(trim "${value-}") ;; + distributionSha256Sum) distributionSha256Sum=$(trim "${value-}") ;; + esac +done <"$scriptDir/.mvn/wrapper/maven-wrapper.properties" +[ -n "${distributionUrl-}" ] || die "cannot read distributionUrl property in $scriptDir/.mvn/wrapper/maven-wrapper.properties" + +case "${distributionUrl##*/}" in +maven-mvnd-*bin.*) + MVN_CMD=mvnd.sh _MVNW_REPO_PATTERN=/maven/mvnd/ + case "${PROCESSOR_ARCHITECTURE-}${PROCESSOR_ARCHITEW6432-}:$(uname -a)" in + *AMD64:CYGWIN* | *AMD64:MINGW*) distributionPlatform=windows-amd64 ;; + :Darwin*x86_64) distributionPlatform=darwin-amd64 ;; + :Darwin*arm64) distributionPlatform=darwin-aarch64 ;; + :Linux*x86_64*) distributionPlatform=linux-amd64 ;; + *) + echo "Cannot detect native platform for mvnd on $(uname)-$(uname -m), use pure java version" >&2 + distributionPlatform=linux-amd64 + ;; + esac + distributionUrl="${distributionUrl%-bin.*}-$distributionPlatform.zip" + ;; +maven-mvnd-*) MVN_CMD=mvnd.sh _MVNW_REPO_PATTERN=/maven/mvnd/ ;; +*) MVN_CMD="mvn${scriptName#mvnw}" _MVNW_REPO_PATTERN=/org/apache/maven/ ;; +esac + +# apply MVNW_REPOURL and calculate MAVEN_HOME +# maven home pattern: ~/.m2/wrapper/dists/{apache-maven-,maven-mvnd--}/ +[ -z "${MVNW_REPOURL-}" ] || distributionUrl="$MVNW_REPOURL$_MVNW_REPO_PATTERN${distributionUrl#*"$_MVNW_REPO_PATTERN"}" +distributionUrlName="${distributionUrl##*/}" +distributionUrlNameMain="${distributionUrlName%.*}" +distributionUrlNameMain="${distributionUrlNameMain%-bin}" +MAVEN_USER_HOME="${MAVEN_USER_HOME:-${HOME}/.m2}" +MAVEN_HOME="${MAVEN_USER_HOME}/wrapper/dists/${distributionUrlNameMain-}/$(hash_string "$distributionUrl")" + +exec_maven() { + unset MVNW_VERBOSE MVNW_USERNAME MVNW_PASSWORD MVNW_REPOURL || : + exec "$MAVEN_HOME/bin/$MVN_CMD" "$@" || die "cannot exec $MAVEN_HOME/bin/$MVN_CMD" +} + +if [ -d "$MAVEN_HOME" ]; then + verbose "found existing MAVEN_HOME at $MAVEN_HOME" + exec_maven "$@" +fi + +case "${distributionUrl-}" in +*?-bin.zip | *?maven-mvnd-?*-?*.zip) ;; +*) die "distributionUrl is not valid, must match *-bin.zip or maven-mvnd-*.zip, but found '${distributionUrl-}'" ;; +esac + +# prepare tmp dir +if TMP_DOWNLOAD_DIR="$(mktemp -d)" && [ -d "$TMP_DOWNLOAD_DIR" ]; then + clean() { rm -rf -- "$TMP_DOWNLOAD_DIR"; } + trap clean HUP INT TERM EXIT +else + die "cannot create temp dir" +fi + +mkdir -p -- "${MAVEN_HOME%/*}" + +# Download and Install Apache Maven +verbose "Couldn't find MAVEN_HOME, downloading and installing it ..." +verbose "Downloading from: $distributionUrl" +verbose "Downloading to: $TMP_DOWNLOAD_DIR/$distributionUrlName" + +# select .zip or .tar.gz +if ! command -v unzip >/dev/null; then + distributionUrl="${distributionUrl%.zip}.tar.gz" + distributionUrlName="${distributionUrl##*/}" +fi + +# verbose opt +__MVNW_QUIET_WGET=--quiet __MVNW_QUIET_CURL=--silent __MVNW_QUIET_UNZIP=-q __MVNW_QUIET_TAR='' +[ "${MVNW_VERBOSE-}" != true ] || __MVNW_QUIET_WGET='' __MVNW_QUIET_CURL='' __MVNW_QUIET_UNZIP='' __MVNW_QUIET_TAR=v + +# normalize http auth +case "${MVNW_PASSWORD:+has-password}" in +'') MVNW_USERNAME='' MVNW_PASSWORD='' ;; +has-password) [ -n "${MVNW_USERNAME-}" ] || MVNW_USERNAME='' MVNW_PASSWORD='' ;; +esac + +if [ -z "${MVNW_USERNAME-}" ] && command -v wget >/dev/null; then + verbose "Found wget ... using wget" + wget ${__MVNW_QUIET_WGET:+"$__MVNW_QUIET_WGET"} "$distributionUrl" -O "$TMP_DOWNLOAD_DIR/$distributionUrlName" || die "wget: Failed to fetch $distributionUrl" +elif [ -z "${MVNW_USERNAME-}" ] && command -v curl >/dev/null; then + verbose "Found curl ... using curl" + curl ${__MVNW_QUIET_CURL:+"$__MVNW_QUIET_CURL"} -f -L -o "$TMP_DOWNLOAD_DIR/$distributionUrlName" "$distributionUrl" || die "curl: Failed to fetch $distributionUrl" +elif set_java_home; then + verbose "Falling back to use Java to download" + javaSource="$TMP_DOWNLOAD_DIR/Downloader.java" + targetZip="$TMP_DOWNLOAD_DIR/$distributionUrlName" + cat >"$javaSource" <<-END + public class Downloader extends java.net.Authenticator + { + protected java.net.PasswordAuthentication getPasswordAuthentication() + { + return new java.net.PasswordAuthentication( System.getenv( "MVNW_USERNAME" ), System.getenv( "MVNW_PASSWORD" ).toCharArray() ); + } + public static void main( String[] args ) throws Exception + { + setDefault( new Downloader() ); + java.nio.file.Files.copy( java.net.URI.create( args[0] ).toURL().openStream(), java.nio.file.Paths.get( args[1] ).toAbsolutePath().normalize() ); + } + } + END + # For Cygwin/MinGW, switch paths to Windows format before running javac and java + verbose " - Compiling Downloader.java ..." + "$(native_path "$JAVACCMD")" "$(native_path "$javaSource")" || die "Failed to compile Downloader.java" + verbose " - Running Downloader.java ..." + "$(native_path "$JAVACMD")" -cp "$(native_path "$TMP_DOWNLOAD_DIR")" Downloader "$distributionUrl" "$(native_path "$targetZip")" +fi + +# If specified, validate the SHA-256 sum of the Maven distribution zip file +if [ -n "${distributionSha256Sum-}" ]; then + distributionSha256Result=false + if [ "$MVN_CMD" = mvnd.sh ]; then + echo "Checksum validation is not supported for maven-mvnd." >&2 + echo "Please disable validation by removing 'distributionSha256Sum' from your maven-wrapper.properties." >&2 + exit 1 + elif command -v sha256sum >/dev/null; then + if echo "$distributionSha256Sum $TMP_DOWNLOAD_DIR/$distributionUrlName" | sha256sum -c - >/dev/null 2>&1; then + distributionSha256Result=true + fi + elif command -v shasum >/dev/null; then + if echo "$distributionSha256Sum $TMP_DOWNLOAD_DIR/$distributionUrlName" | shasum -a 256 -c >/dev/null 2>&1; then + distributionSha256Result=true + fi + else + echo "Checksum validation was requested but neither 'sha256sum' or 'shasum' are available." >&2 + echo "Please install either command, or disable validation by removing 'distributionSha256Sum' from your maven-wrapper.properties." >&2 + exit 1 + fi + if [ $distributionSha256Result = false ]; then + echo "Error: Failed to validate Maven distribution SHA-256, your Maven distribution might be compromised." >&2 + echo "If you updated your Maven version, you need to update the specified distributionSha256Sum property." >&2 + exit 1 + fi +fi + +# unzip and move +if command -v unzip >/dev/null; then + unzip ${__MVNW_QUIET_UNZIP:+"$__MVNW_QUIET_UNZIP"} "$TMP_DOWNLOAD_DIR/$distributionUrlName" -d "$TMP_DOWNLOAD_DIR" || die "failed to unzip" +else + tar xzf${__MVNW_QUIET_TAR:+"$__MVNW_QUIET_TAR"} "$TMP_DOWNLOAD_DIR/$distributionUrlName" -C "$TMP_DOWNLOAD_DIR" || die "failed to untar" +fi + +# Find the actual extracted directory name (handles snapshots where filename != directory name) +actualDistributionDir="" + +# First try the expected directory name (for regular distributions) +if [ -d "$TMP_DOWNLOAD_DIR/$distributionUrlNameMain" ]; then + if [ -f "$TMP_DOWNLOAD_DIR/$distributionUrlNameMain/bin/$MVN_CMD" ]; then + actualDistributionDir="$distributionUrlNameMain" + fi +fi + +# If not found, search for any directory with the Maven executable (for snapshots) +if [ -z "$actualDistributionDir" ]; then + # enable globbing to iterate over items + set +f + for dir in "$TMP_DOWNLOAD_DIR"/*; do + if [ -d "$dir" ]; then + if [ -f "$dir/bin/$MVN_CMD" ]; then + actualDistributionDir="$(basename "$dir")" + break + fi + fi + done + set -f +fi + +if [ -z "$actualDistributionDir" ]; then + verbose "Contents of $TMP_DOWNLOAD_DIR:" + verbose "$(ls -la "$TMP_DOWNLOAD_DIR")" + die "Could not find Maven distribution directory in extracted archive" +fi + +verbose "Found extracted Maven distribution directory: $actualDistributionDir" +printf %s\\n "$distributionUrl" >"$TMP_DOWNLOAD_DIR/$actualDistributionDir/mvnw.url" +mv -- "$TMP_DOWNLOAD_DIR/$actualDistributionDir" "$MAVEN_HOME" || [ -d "$MAVEN_HOME" ] || die "fail to move MAVEN_HOME" + +clean || : +exec_maven "$@" diff --git a/mvnw.cmd b/mvnw.cmd new file mode 100644 index 0000000..5761d94 --- /dev/null +++ b/mvnw.cmd @@ -0,0 +1,189 @@ +<# : batch portion +@REM ---------------------------------------------------------------------------- +@REM Licensed to the Apache Software Foundation (ASF) under one +@REM or more contributor license agreements. See the NOTICE file +@REM distributed with this work for additional information +@REM regarding copyright ownership. The ASF licenses this file +@REM to you under the Apache License, Version 2.0 (the +@REM "License"); you may not use this file except in compliance +@REM with the License. You may obtain a copy of the License at +@REM +@REM http://www.apache.org/licenses/LICENSE-2.0 +@REM +@REM Unless required by applicable law or agreed to in writing, +@REM software distributed under the License is distributed on an +@REM "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +@REM KIND, either express or implied. See the License for the +@REM specific language governing permissions and limitations +@REM under the License. +@REM ---------------------------------------------------------------------------- + +@REM ---------------------------------------------------------------------------- +@REM Apache Maven Wrapper startup batch script, version 3.3.4 +@REM +@REM Optional ENV vars +@REM MVNW_REPOURL - repo url base for downloading maven distribution +@REM MVNW_USERNAME/MVNW_PASSWORD - user and password for downloading maven +@REM MVNW_VERBOSE - true: enable verbose log; others: silence the output +@REM ---------------------------------------------------------------------------- + +@IF "%__MVNW_ARG0_NAME__%"=="" (SET __MVNW_ARG0_NAME__=%~nx0) +@SET __MVNW_CMD__= +@SET __MVNW_ERROR__= +@SET __MVNW_PSMODULEP_SAVE=%PSModulePath% +@SET PSModulePath= +@FOR /F "usebackq tokens=1* delims==" %%A IN (`powershell -noprofile "& {$scriptDir='%~dp0'; $script='%__MVNW_ARG0_NAME__%'; icm -ScriptBlock ([Scriptblock]::Create((Get-Content -Raw '%~f0'))) -NoNewScope}"`) DO @( + IF "%%A"=="MVN_CMD" (set __MVNW_CMD__=%%B) ELSE IF "%%B"=="" (echo %%A) ELSE (echo %%A=%%B) +) +@SET PSModulePath=%__MVNW_PSMODULEP_SAVE% +@SET __MVNW_PSMODULEP_SAVE= +@SET __MVNW_ARG0_NAME__= +@SET MVNW_USERNAME= +@SET MVNW_PASSWORD= +@IF NOT "%__MVNW_CMD__%"=="" ("%__MVNW_CMD__%" %*) +@echo Cannot start maven from wrapper >&2 && exit /b 1 +@GOTO :EOF +: end batch / begin powershell #> + +$ErrorActionPreference = "Stop" +if ($env:MVNW_VERBOSE -eq "true") { + $VerbosePreference = "Continue" +} + +# calculate distributionUrl, requires .mvn/wrapper/maven-wrapper.properties +$distributionUrl = (Get-Content -Raw "$scriptDir/.mvn/wrapper/maven-wrapper.properties" | ConvertFrom-StringData).distributionUrl +if (!$distributionUrl) { + Write-Error "cannot read distributionUrl property in $scriptDir/.mvn/wrapper/maven-wrapper.properties" +} + +switch -wildcard -casesensitive ( $($distributionUrl -replace '^.*/','') ) { + "maven-mvnd-*" { + $USE_MVND = $true + $distributionUrl = $distributionUrl -replace '-bin\.[^.]*$',"-windows-amd64.zip" + $MVN_CMD = "mvnd.cmd" + break + } + default { + $USE_MVND = $false + $MVN_CMD = $script -replace '^mvnw','mvn' + break + } +} + +# apply MVNW_REPOURL and calculate MAVEN_HOME +# maven home pattern: ~/.m2/wrapper/dists/{apache-maven-,maven-mvnd--}/ +if ($env:MVNW_REPOURL) { + $MVNW_REPO_PATTERN = if ($USE_MVND -eq $False) { "/org/apache/maven/" } else { "/maven/mvnd/" } + $distributionUrl = "$env:MVNW_REPOURL$MVNW_REPO_PATTERN$($distributionUrl -replace "^.*$MVNW_REPO_PATTERN",'')" +} +$distributionUrlName = $distributionUrl -replace '^.*/','' +$distributionUrlNameMain = $distributionUrlName -replace '\.[^.]*$','' -replace '-bin$','' + +$MAVEN_M2_PATH = "$HOME/.m2" +if ($env:MAVEN_USER_HOME) { + $MAVEN_M2_PATH = "$env:MAVEN_USER_HOME" +} + +if (-not (Test-Path -Path $MAVEN_M2_PATH)) { + New-Item -Path $MAVEN_M2_PATH -ItemType Directory | Out-Null +} + +$MAVEN_WRAPPER_DISTS = $null +if ((Get-Item $MAVEN_M2_PATH).Target[0] -eq $null) { + $MAVEN_WRAPPER_DISTS = "$MAVEN_M2_PATH/wrapper/dists" +} else { + $MAVEN_WRAPPER_DISTS = (Get-Item $MAVEN_M2_PATH).Target[0] + "/wrapper/dists" +} + +$MAVEN_HOME_PARENT = "$MAVEN_WRAPPER_DISTS/$distributionUrlNameMain" +$MAVEN_HOME_NAME = ([System.Security.Cryptography.SHA256]::Create().ComputeHash([byte[]][char[]]$distributionUrl) | ForEach-Object {$_.ToString("x2")}) -join '' +$MAVEN_HOME = "$MAVEN_HOME_PARENT/$MAVEN_HOME_NAME" + +if (Test-Path -Path "$MAVEN_HOME" -PathType Container) { + Write-Verbose "found existing MAVEN_HOME at $MAVEN_HOME" + Write-Output "MVN_CMD=$MAVEN_HOME/bin/$MVN_CMD" + exit $? +} + +if (! $distributionUrlNameMain -or ($distributionUrlName -eq $distributionUrlNameMain)) { + Write-Error "distributionUrl is not valid, must end with *-bin.zip, but found $distributionUrl" +} + +# prepare tmp dir +$TMP_DOWNLOAD_DIR_HOLDER = New-TemporaryFile +$TMP_DOWNLOAD_DIR = New-Item -Itemtype Directory -Path "$TMP_DOWNLOAD_DIR_HOLDER.dir" +$TMP_DOWNLOAD_DIR_HOLDER.Delete() | Out-Null +trap { + if ($TMP_DOWNLOAD_DIR.Exists) { + try { Remove-Item $TMP_DOWNLOAD_DIR -Recurse -Force | Out-Null } + catch { Write-Warning "Cannot remove $TMP_DOWNLOAD_DIR" } + } +} + +New-Item -Itemtype Directory -Path "$MAVEN_HOME_PARENT" -Force | Out-Null + +# Download and Install Apache Maven +Write-Verbose "Couldn't find MAVEN_HOME, downloading and installing it ..." +Write-Verbose "Downloading from: $distributionUrl" +Write-Verbose "Downloading to: $TMP_DOWNLOAD_DIR/$distributionUrlName" + +$webclient = New-Object System.Net.WebClient +if ($env:MVNW_USERNAME -and $env:MVNW_PASSWORD) { + $webclient.Credentials = New-Object System.Net.NetworkCredential($env:MVNW_USERNAME, $env:MVNW_PASSWORD) +} +[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 +$webclient.DownloadFile($distributionUrl, "$TMP_DOWNLOAD_DIR/$distributionUrlName") | Out-Null + +# If specified, validate the SHA-256 sum of the Maven distribution zip file +$distributionSha256Sum = (Get-Content -Raw "$scriptDir/.mvn/wrapper/maven-wrapper.properties" | ConvertFrom-StringData).distributionSha256Sum +if ($distributionSha256Sum) { + if ($USE_MVND) { + Write-Error "Checksum validation is not supported for maven-mvnd. `nPlease disable validation by removing 'distributionSha256Sum' from your maven-wrapper.properties." + } + Import-Module $PSHOME\Modules\Microsoft.PowerShell.Utility -Function Get-FileHash + if ((Get-FileHash "$TMP_DOWNLOAD_DIR/$distributionUrlName" -Algorithm SHA256).Hash.ToLower() -ne $distributionSha256Sum) { + Write-Error "Error: Failed to validate Maven distribution SHA-256, your Maven distribution might be compromised. If you updated your Maven version, you need to update the specified distributionSha256Sum property." + } +} + +# unzip and move +Expand-Archive "$TMP_DOWNLOAD_DIR/$distributionUrlName" -DestinationPath "$TMP_DOWNLOAD_DIR" | Out-Null + +# Find the actual extracted directory name (handles snapshots where filename != directory name) +$actualDistributionDir = "" + +# First try the expected directory name (for regular distributions) +$expectedPath = Join-Path "$TMP_DOWNLOAD_DIR" "$distributionUrlNameMain" +$expectedMvnPath = Join-Path "$expectedPath" "bin/$MVN_CMD" +if ((Test-Path -Path $expectedPath -PathType Container) -and (Test-Path -Path $expectedMvnPath -PathType Leaf)) { + $actualDistributionDir = $distributionUrlNameMain +} + +# If not found, search for any directory with the Maven executable (for snapshots) +if (!$actualDistributionDir) { + Get-ChildItem -Path "$TMP_DOWNLOAD_DIR" -Directory | ForEach-Object { + $testPath = Join-Path $_.FullName "bin/$MVN_CMD" + if (Test-Path -Path $testPath -PathType Leaf) { + $actualDistributionDir = $_.Name + } + } +} + +if (!$actualDistributionDir) { + Write-Error "Could not find Maven distribution directory in extracted archive" +} + +Write-Verbose "Found extracted Maven distribution directory: $actualDistributionDir" +Rename-Item -Path "$TMP_DOWNLOAD_DIR/$actualDistributionDir" -NewName $MAVEN_HOME_NAME | Out-Null +try { + Move-Item -Path "$TMP_DOWNLOAD_DIR/$MAVEN_HOME_NAME" -Destination $MAVEN_HOME_PARENT | Out-Null +} catch { + if (! (Test-Path -Path "$MAVEN_HOME" -PathType Container)) { + Write-Error "fail to move MAVEN_HOME" + } +} finally { + try { Remove-Item $TMP_DOWNLOAD_DIR -Recurse -Force | Out-Null } + catch { Write-Warning "Cannot remove $TMP_DOWNLOAD_DIR" } +} + +Write-Output "MVN_CMD=$MAVEN_HOME/bin/$MVN_CMD" diff --git a/native/Cargo.lock b/native/Cargo.lock new file mode 100644 index 0000000..5ebef8e --- /dev/null +++ b/native/Cargo.lock @@ -0,0 +1,3370 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "adler2" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" + +[[package]] +name = "ahash" +version = "0.8.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75" +dependencies = [ + "cfg-if", + "const-random", + "getrandom 0.3.4", + "once_cell", + "version_check", + "zerocopy", +] + +[[package]] +name = "aho-corasick" +version = "1.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ddd31a130427c27518df266943a5308ed92d4b226cc639f5a8f1002816174301" +dependencies = [ + "memchr", +] + +[[package]] +name = "alloc-no-stdlib" +version = "2.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cc7bb162ec39d46ab1ca8c77bf72e890535becd1751bb45f64c597edb4c8c6b3" + +[[package]] +name = "alloc-stdlib" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "94fb8275041c72129eb51b7d0322c29b8387a0386127718b096429201a5d6ece" +dependencies = [ + "alloc-no-stdlib", +] + +[[package]] +name = "allocator-api2" +version = "0.2.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "683d7910e743518b0e34f1186f92494becacb047c7b6bf616c96772180fef923" + +[[package]] +name = "android_system_properties" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "819e7219dbd41043ac279b19830f2efc897156490d7fd6ea916720117ee66311" +dependencies = [ + "libc", +] + +[[package]] +name = "anyhow" +version = "1.0.102" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7f202df86484c868dbad7eaa557ef785d5c66295e41b460ef922eca0723b842c" + +[[package]] +name = "ar_archive_writer" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7eb93bbb63b9c227414f6eb3a0adfddca591a8ce1e9b60661bb08969b87e340b" +dependencies = [ + "object", +] + +[[package]] +name = "arrayref" +version = "0.3.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "76a2e8124351fda1ef8aaaa3bbd7ebbcb486bbcd4225aca0aa0d84bb2db8fecb" + +[[package]] +name = "arrayvec" +version = "0.7.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50" + +[[package]] +name = "arrow" +version = "58.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "607e64bb911ee4f90483e044fe78f175989148c2892e659a2cd25429e782ec54" +dependencies = [ + "arrow-arith", + "arrow-array", + "arrow-buffer", + "arrow-cast", + "arrow-csv", + "arrow-data", + "arrow-ipc", + "arrow-json", + "arrow-ord", + "arrow-row", + "arrow-schema", + "arrow-select", + "arrow-string", +] + +[[package]] +name = "arrow-arith" +version = "58.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e754319ed8a85d817fe7adf183227e0b5308b82790a737b426c1124626b48118" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-schema", + "chrono", + "num-traits", +] + +[[package]] +name = "arrow-array" +version = "58.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "841321891f247aa86c6112c80d83d89cb36e0addd020fa2425085b8eb6c3f579" +dependencies = [ + "ahash", + "arrow-buffer", + "arrow-data", + "arrow-schema", + "chrono", + "chrono-tz", + "half", + "hashbrown 0.17.1", + "num-complex", + "num-integer", + "num-traits", +] + +[[package]] +name = "arrow-buffer" +version = "58.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f955dfb73fae000425f49c8226d2044dab60fb7ad4af1e24f961756354d996c9" +dependencies = [ + "bytes", + "half", + "num-bigint", + "num-traits", +] + +[[package]] +name = "arrow-cast" +version = "58.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ca5e686972523798f76bef355145bc1ae25a84c731e650268d31ab763c701663" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-ord", + "arrow-schema", + "arrow-select", + "atoi", + "base64", + "chrono", + "comfy-table", + "half", + "lexical-core", + "num-traits", + "ryu", +] + +[[package]] +name = "arrow-csv" +version = "58.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "86c276756867fc8186ec380c72c290e6e3b23a1d4fb05df6b1d62d2e62666d48" +dependencies = [ + "arrow-array", + "arrow-cast", + "arrow-schema", + "chrono", + "csv", + "csv-core", + "regex", +] + +[[package]] +name = "arrow-data" +version = "58.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "db3b5846209775b6dc8056d77ff9a032b27043383dd5488abd0b663e265b9373" +dependencies = [ + "arrow-buffer", + "arrow-schema", + "half", + "num-integer", + "num-traits", +] + +[[package]] +name = "arrow-ipc" +version = "58.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fd8907ddd8f9fbabf91ec2c85c1d81fe2874e336d2443eb36373595e28b98dd5" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-schema", + "arrow-select", + "flatbuffers", + "lz4_flex", + "zstd", +] + +[[package]] +name = "arrow-json" +version = "58.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f4518c59acc501f10d7dcae397fe12b8db3d81bc7de94456f8a58f9165d6f502" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-cast", + "arrow-ord", + "arrow-schema", + "arrow-select", + "chrono", + "half", + "indexmap", + "itoa", + "lexical-core", + "memchr", + "num-traits", + "ryu", + "serde_core", + "serde_json", + "simdutf8", +] + +[[package]] +name = "arrow-ord" +version = "58.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "efa70d9d6b1356f1fb9f1f651b84a725b7e0abb93f188cf7d31f14abfa2f2e6f" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-schema", + "arrow-select", +] + +[[package]] +name = "arrow-row" +version = "58.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "faec88a945338192beffbbd4be0def70135422930caa244ac3cec0cd213b26b4" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-schema", + "half", +] + +[[package]] +name = "arrow-schema" +version = "58.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "18aa020f6bc8e5201dcd2d4b7f98c68f8a410ef37128263243e6ff2a47a67d4f" +dependencies = [ + "bitflags", + "serde_core", + "serde_json", +] + +[[package]] +name = "arrow-select" +version = "58.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a657ab5132e9c8ca3b24eb15a823d0ced38017fe3930ff50167466b02e2d592c" +dependencies = [ + "ahash", + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-schema", + "num-traits", +] + +[[package]] +name = "arrow-string" +version = "58.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f6de2efbbd1a9f9780ceb8d1ff5d20421b35863b361e3386b4f571f1fc69fcb8" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-schema", + "arrow-select", + "memchr", + "num-traits", + "regex", + "regex-syntax", +] + +[[package]] +name = "async-compression" +version = "0.4.42" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e79b3f8a79cccc2898f31920fc69f304859b3bd567490f75ebf51ae1c792a9ac" +dependencies = [ + "compression-codecs", + "compression-core", + "pin-project-lite", + "tokio", +] + +[[package]] +name = "async-trait" +version = "0.1.89" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9035ad2d096bed7955a320ee7e2230574d28fd3c3a0f186cbea1ff3c7eed5dbb" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "atoi" +version = "2.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f28d99ec8bfea296261ca1af174f24225171fea9664ba9003cbebee704810528" +dependencies = [ + "num-traits", +] + +[[package]] +name = "autocfg" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8" + +[[package]] +name = "base64" +version = "0.22.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" + +[[package]] +name = "bigdecimal" +version = "0.4.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4d6867f1565b3aad85681f1015055b087fcfd840d6aeee6eee7f2da317603695" +dependencies = [ + "autocfg", + "libm", + "num-bigint", + "num-integer", + "num-traits", +] + +[[package]] +name = "bitflags" +version = "2.11.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c4512299f36f043ab09a583e57bceb5a5aab7a73db1805848e8fef3c9e8c78b3" + +[[package]] +name = "blake2" +version = "0.10.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "46502ad458c9a52b69d4d4d32775c788b7a1b85e8bc9d482d92250fc0e3f8efe" +dependencies = [ + "digest", +] + +[[package]] +name = "blake3" +version = "1.8.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0aa83c34e62843d924f905e0f5c866eb1dd6545fc4d719e803d9ba6030371fce" +dependencies = [ + "arrayref", + "arrayvec", + "cc", + "cfg-if", + "constant_time_eq", + "cpufeatures 0.3.0", +] + +[[package]] +name = "block-buffer" +version = "0.10.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71" +dependencies = [ + "generic-array", +] + +[[package]] +name = "brotli" +version = "8.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4bd8b9603c7aa97359dbd97ecf258968c95f3adddd6db2f7e7a5bef101c84560" +dependencies = [ + "alloc-no-stdlib", + "alloc-stdlib", + "brotli-decompressor", +] + +[[package]] +name = "brotli-decompressor" +version = "5.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "874bb8112abecc98cbd6d81ea4fa7e94fb9449648c93cc89aa40c81c24d7de03" +dependencies = [ + "alloc-no-stdlib", + "alloc-stdlib", +] + +[[package]] +name = "bumpalo" +version = "3.20.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5d20789868f4b01b2f2caec9f5c4e0213b41e3e5702a50157d699ae31ced2fcb" + +[[package]] +name = "byteorder" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" + +[[package]] +name = "bytes" +version = "1.11.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e748733b7cbc798e1434b6ac524f0c1ff2ab456fe201501e6497c8417a4fc33" + +[[package]] +name = "bzip2" +version = "0.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f3a53fac24f34a81bc9954b5d6cfce0c21e18ec6959f44f56e8e90e4bb7c346c" +dependencies = [ + "libbz2-rs-sys", +] + +[[package]] +name = "cc" +version = "1.2.62" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a1dce859f0832a7d088c4f1119888ab94ef4b5d6795d1ce05afb7fe159d79f98" +dependencies = [ + "find-msvc-tools", + "jobserver", + "libc", + "shlex", +] + +[[package]] +name = "cesu8" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6d43a04d8753f35258c91f8ec639f792891f748a1edbd759cf1dcea3382ad83c" + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "chrono" +version = "0.4.44" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c673075a2e0e5f4a1dde27ce9dee1ea4558c7ffe648f576438a20ca1d2acc4b0" +dependencies = [ + "iana-time-zone", + "num-traits", + "windows-link", +] + +[[package]] +name = "chrono-tz" +version = "0.10.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6139a8597ed92cf816dfb33f5dd6cf0bb93a6adc938f11039f371bc5bcd26c3" +dependencies = [ + "chrono", + "phf", +] + +[[package]] +name = "combine" +version = "4.6.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ba5a308b75df32fe02788e748662718f03fde005016435c444eea572398219fd" +dependencies = [ + "bytes", + "memchr", +] + +[[package]] +name = "comfy-table" +version = "7.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "958c5d6ecf1f214b4c2bbbbf6ab9523a864bd136dcf71a7e8904799acfe1ad47" +dependencies = [ + "unicode-segmentation", + "unicode-width", +] + +[[package]] +name = "compression-codecs" +version = "0.4.38" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ce2548391e9c1929c21bf6aa2680af86fe4c1b33e6cea9ac1cfeec0bd11218cf" +dependencies = [ + "bzip2", + "compression-core", + "flate2", + "liblzma", + "memchr", + "zstd", + "zstd-safe", +] + +[[package]] +name = "compression-core" +version = "0.4.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cc14f565cf027a105f7a44ccf9e5b424348421a1d8952a8fc9d499d313107789" + +[[package]] +name = "const-random" +version = "0.1.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "87e00182fe74b066627d63b85fd550ac2998d4b0bd86bfed477a0ae4c7c71359" +dependencies = [ + "const-random-macro", +] + +[[package]] +name = "const-random-macro" +version = "0.1.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f9d839f2a20b0aee515dc581a6172f2321f96cab76c1a38a4c584a194955390e" +dependencies = [ + "getrandom 0.2.17", + "once_cell", + "tiny-keccak", +] + +[[package]] +name = "constant_time_eq" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3d52eff69cd5e647efe296129160853a42795992097e8af39800e1060caeea9b" + +[[package]] +name = "core-foundation-sys" +version = "0.8.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" + +[[package]] +name = "cpufeatures" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280" +dependencies = [ + "libc", +] + +[[package]] +name = "cpufeatures" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201" +dependencies = [ + "libc", +] + +[[package]] +name = "crc32fast" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9481c1c90cbf2ac953f07c8d4a58aa3945c425b7185c9154d67a65e4230da511" +dependencies = [ + "cfg-if", +] + +[[package]] +name = "crossbeam-utils" +version = "0.8.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d0a5c400df2834b80a4c3327b3aad3a4c4cd4de0629063962b03235697506a28" + +[[package]] +name = "crunchy" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "460fbee9c2c2f33933d720630a6a0bac33ba7053db5344fac858d4b8952d77d5" + +[[package]] +name = "crypto-common" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" +dependencies = [ + "generic-array", + "typenum", +] + +[[package]] +name = "csv" +version = "1.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "52cd9d68cf7efc6ddfaaee42e7288d3a99d613d4b50f76ce9827ae0c6e14f938" +dependencies = [ + "csv-core", + "itoa", + "ryu", + "serde_core", +] + +[[package]] +name = "csv-core" +version = "0.1.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "704a3c26996a80471189265814dbc2c257598b96b8a7feae2d31ace646bb9782" +dependencies = [ + "memchr", +] + +[[package]] +name = "dashmap" +version = "6.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5041cc499144891f3790297212f32a74fb938e5136a14943f338ef9e0ae276cf" +dependencies = [ + "cfg-if", + "crossbeam-utils", + "hashbrown 0.14.5", + "lock_api", + "once_cell", + "parking_lot_core", +] + +[[package]] +name = "datafusion" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "93db0e623840612f7f2cd757f7e8a8922064192363732c88692e0870016e141b" +dependencies = [ + "arrow", + "arrow-schema", + "async-trait", + "bytes", + "bzip2", + "chrono", + "datafusion-catalog", + "datafusion-catalog-listing", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-datasource", + "datafusion-datasource-arrow", + "datafusion-datasource-csv", + "datafusion-datasource-json", + "datafusion-datasource-parquet", + "datafusion-execution", + "datafusion-expr", + "datafusion-expr-common", + "datafusion-functions", + "datafusion-functions-aggregate", + "datafusion-functions-nested", + "datafusion-functions-table", + "datafusion-functions-window", + "datafusion-optimizer", + "datafusion-physical-expr", + "datafusion-physical-expr-adapter", + "datafusion-physical-expr-common", + "datafusion-physical-optimizer", + "datafusion-physical-plan", + "datafusion-session", + "datafusion-sql", + "flate2", + "futures", + "itertools", + "liblzma", + "log", + "object_store", + "parking_lot", + "parquet", + "rand", + "regex", + "sqlparser", + "tempfile", + "tokio", + "url", + "uuid", + "zstd", +] + +[[package]] +name = "datafusion-catalog" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "37cefde60b26a7f4ff61e9d2ff2833322f91df2b568d7238afe67bde5bdffb66" +dependencies = [ + "arrow", + "async-trait", + "dashmap", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr", + "datafusion-physical-plan", + "datafusion-session", + "futures", + "itertools", + "log", + "object_store", + "parking_lot", + "tokio", +] + +[[package]] +name = "datafusion-catalog-listing" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "17e112307715d6a7a331111a4c2330ff54bc237183511c319e3708a4cff431fb" +dependencies = [ + "arrow", + "async-trait", + "datafusion-catalog", + "datafusion-common", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr", + "datafusion-physical-expr-adapter", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "futures", + "itertools", + "log", + "object_store", +] + +[[package]] +name = "datafusion-common" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d72a11ca44a95e1081870d3abb80c717496e8a7acb467a1d3e932bb636af5cc2" +dependencies = [ + "ahash", + "arrow", + "arrow-ipc", + "chrono", + "half", + "hashbrown 0.16.1", + "indexmap", + "itertools", + "libc", + "log", + "object_store", + "parquet", + "paste", + "recursive", + "sqlparser", + "tokio", + "web-time", +] + +[[package]] +name = "datafusion-common-runtime" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "89f4afaed29670ec4fd6053643adc749fe3f4bc9d1ce1b8c5679b22c67d12def" +dependencies = [ + "futures", + "log", + "tokio", +] + +[[package]] +name = "datafusion-datasource" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e9fb386e1691355355a96419978a0022b7947b44d4a24a6ea99f00b6b485cbb6" +dependencies = [ + "arrow", + "async-compression", + "async-trait", + "bytes", + "bzip2", + "chrono", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr", + "datafusion-physical-expr-adapter", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-session", + "flate2", + "futures", + "glob", + "itertools", + "liblzma", + "log", + "object_store", + "rand", + "tokio", + "tokio-util", + "url", + "zstd", +] + +[[package]] +name = "datafusion-datasource-arrow" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ffa6c52cfed0734c5f93754d1c0175f558175248bf686c944fb05c373e5fc096" +dependencies = [ + "arrow", + "arrow-ipc", + "async-trait", + "bytes", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-session", + "futures", + "itertools", + "object_store", + "tokio", +] + +[[package]] +name = "datafusion-datasource-csv" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "503f29e0582c1fc189578d665ff57d9300da1f80c282777d7eb67bb79fb8cdca" +dependencies = [ + "arrow", + "async-trait", + "bytes", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-session", + "futures", + "object_store", + "regex", + "tokio", +] + +[[package]] +name = "datafusion-datasource-json" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e33804749abc8d0c8cb7473228483cb8070e524c6f6086ee1b85a64debe2b3d2" +dependencies = [ + "arrow", + "async-trait", + "bytes", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-session", + "futures", + "object_store", + "serde_json", + "tokio", + "tokio-stream", +] + +[[package]] +name = "datafusion-datasource-parquet" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a8e0365e0e08e8ff94d912f0ababcf9065a1a304018ba90b1fc83c855b4997" +dependencies = [ + "arrow", + "async-trait", + "bytes", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", + "datafusion-functions-aggregate-common", + "datafusion-physical-expr", + "datafusion-physical-expr-adapter", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-pruning", + "datafusion-session", + "futures", + "itertools", + "log", + "object_store", + "parking_lot", + "parquet", + "tokio", +] + +[[package]] +name = "datafusion-doc" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8de6ac0df1662b9148ad3c987978b32cbec7c772f199b1d53520c8fa764a87ee" + +[[package]] +name = "datafusion-execution" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c03c7fbdaefcca4ef6ffe425a5fc2325763bfb426599bb0bf4536466efabe709" +dependencies = [ + "arrow", + "arrow-buffer", + "async-trait", + "chrono", + "dashmap", + "datafusion-common", + "datafusion-expr", + "datafusion-physical-expr-common", + "futures", + "log", + "object_store", + "parking_lot", + "rand", + "tempfile", + "url", +] + +[[package]] +name = "datafusion-expr" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "574b9b6977fedbd2a611cbff12e5caf90f31640ad9dc5870f152836d94bad0dd" +dependencies = [ + "arrow", + "async-trait", + "chrono", + "datafusion-common", + "datafusion-doc", + "datafusion-expr-common", + "datafusion-functions-aggregate-common", + "datafusion-functions-window-common", + "datafusion-physical-expr-common", + "indexmap", + "itertools", + "paste", + "recursive", + "serde_json", + "sqlparser", +] + +[[package]] +name = "datafusion-expr-common" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7d7c3adf3db8bf61e92eb90cb659c8e8b734593a8f7c8e12a843c7ddba24b87e" +dependencies = [ + "arrow", + "datafusion-common", + "indexmap", + "itertools", + "paste", +] + +[[package]] +name = "datafusion-functions" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f28aa4e10384e782774b10e72aca4d93ef7b31aa653095d9d4536b0a3dbc51b6" +dependencies = [ + "arrow", + "arrow-buffer", + "base64", + "blake2", + "blake3", + "chrono", + "chrono-tz", + "datafusion-common", + "datafusion-doc", + "datafusion-execution", + "datafusion-expr", + "datafusion-expr-common", + "datafusion-macros", + "hex", + "itertools", + "log", + "md-5", + "memchr", + "num-traits", + "rand", + "regex", + "sha2", + "unicode-segmentation", + "uuid", +] + +[[package]] +name = "datafusion-functions-aggregate" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "00aa6217e56098ba84e0a338176fe52f0a84cca398021512c6c8c5eff806d0ad" +dependencies = [ + "ahash", + "arrow", + "datafusion-common", + "datafusion-doc", + "datafusion-execution", + "datafusion-expr", + "datafusion-functions-aggregate-common", + "datafusion-macros", + "datafusion-physical-expr", + "datafusion-physical-expr-common", + "half", + "log", + "num-traits", + "paste", +] + +[[package]] +name = "datafusion-functions-aggregate-common" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b511250349407db7c43832ab2de63f5557b19a20dfd236b39ca2c04468b50d47" +dependencies = [ + "ahash", + "arrow", + "datafusion-common", + "datafusion-expr-common", + "datafusion-physical-expr-common", +] + +[[package]] +name = "datafusion-functions-nested" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ef13a858e20d50f0a9bb5e96e7ac82b4e7597f247515bccca4fdd2992df0212a" +dependencies = [ + "arrow", + "arrow-ord", + "datafusion-common", + "datafusion-doc", + "datafusion-execution", + "datafusion-expr", + "datafusion-expr-common", + "datafusion-functions", + "datafusion-functions-aggregate", + "datafusion-functions-aggregate-common", + "datafusion-macros", + "datafusion-physical-expr-common", + "hashbrown 0.16.1", + "itertools", + "itoa", + "log", + "paste", +] + +[[package]] +name = "datafusion-functions-table" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b40d3f5bbb3905f9ccb1ce9485a9595c77b69758a7c24d3ba79e334ff51e7e" +dependencies = [ + "arrow", + "async-trait", + "datafusion-catalog", + "datafusion-common", + "datafusion-expr", + "datafusion-physical-plan", + "parking_lot", + "paste", +] + +[[package]] +name = "datafusion-functions-window" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d4e88ec9d57c9b685d02f58bfee7be62d72610430ddcedb82a08e5d9925dbfb6" +dependencies = [ + "arrow", + "datafusion-common", + "datafusion-doc", + "datafusion-expr", + "datafusion-functions-window-common", + "datafusion-macros", + "datafusion-physical-expr", + "datafusion-physical-expr-common", + "log", + "paste", +] + +[[package]] +name = "datafusion-functions-window-common" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8307bb93519b1a91913723a1130cfafeee3f72200d870d88e91a6fc5470ede5c" +dependencies = [ + "datafusion-common", + "datafusion-physical-expr-common", +] + +[[package]] +name = "datafusion-jni" +version = "0.1.0" +dependencies = [ + "arrow", + "datafusion", + "jni", + "tokio", +] + +[[package]] +name = "datafusion-macros" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2e367e6a71051d0ebdd29b2f85d12059b38b1d1f172c6906e80016da662226bd" +dependencies = [ + "datafusion-doc", + "quote", + "syn", +] + +[[package]] +name = "datafusion-optimizer" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e929015451a67f77d9d8b727b2bf3a40c4445fdef6cdc53281d7d97c76888ace" +dependencies = [ + "arrow", + "chrono", + "datafusion-common", + "datafusion-expr", + "datafusion-expr-common", + "datafusion-physical-expr", + "indexmap", + "itertools", + "log", + "recursive", + "regex", + "regex-syntax", +] + +[[package]] +name = "datafusion-physical-expr" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4b1e68aba7a4b350401cfdf25a3d6f989ad898a7410164afe9ca52080244cb59" +dependencies = [ + "ahash", + "arrow", + "datafusion-common", + "datafusion-expr", + "datafusion-expr-common", + "datafusion-functions-aggregate-common", + "datafusion-physical-expr-common", + "half", + "hashbrown 0.16.1", + "indexmap", + "itertools", + "parking_lot", + "paste", + "petgraph", + "recursive", + "tokio", +] + +[[package]] +name = "datafusion-physical-expr-adapter" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ea22315f33cf2e0adc104e8ec42e285f6ed93998d565c65e82fec6a9ee9f9db4" +dependencies = [ + "arrow", + "datafusion-common", + "datafusion-expr", + "datafusion-functions", + "datafusion-physical-expr", + "datafusion-physical-expr-common", + "itertools", +] + +[[package]] +name = "datafusion-physical-expr-common" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b04b45ea8ad3ac2d78f2ea2a76053e06591c9629c7a603eda16c10649ecf4362" +dependencies = [ + "ahash", + "arrow", + "chrono", + "datafusion-common", + "datafusion-expr-common", + "hashbrown 0.16.1", + "indexmap", + "itertools", + "parking_lot", +] + +[[package]] +name = "datafusion-physical-optimizer" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7cb13397809a425918f608dfe8653f332015a3e330004ab191b4404187238b95" +dependencies = [ + "arrow", + "datafusion-common", + "datafusion-execution", + "datafusion-expr", + "datafusion-expr-common", + "datafusion-physical-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-pruning", + "itertools", + "recursive", +] + +[[package]] +name = "datafusion-physical-plan" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5edc023675791af9d5fb4cc4c24abf5f7bd3bd4dcf9e5bd90ea1eff6976dcc79" +dependencies = [ + "ahash", + "arrow", + "arrow-ord", + "arrow-schema", + "async-trait", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-execution", + "datafusion-expr", + "datafusion-functions", + "datafusion-functions-aggregate-common", + "datafusion-functions-window-common", + "datafusion-physical-expr", + "datafusion-physical-expr-common", + "futures", + "half", + "hashbrown 0.16.1", + "indexmap", + "itertools", + "log", + "num-traits", + "parking_lot", + "pin-project-lite", + "tokio", +] + +[[package]] +name = "datafusion-pruning" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ac8c76860e355616555081cab5968cec1af7a80701ff374510860bcd567e365a" +dependencies = [ + "arrow", + "datafusion-common", + "datafusion-datasource", + "datafusion-expr-common", + "datafusion-physical-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "itertools", + "log", +] + +[[package]] +name = "datafusion-session" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5412111aa48e2424ba926112e192f7a6b7e4ccb450145d25ce5ede9f19dc491e" +dependencies = [ + "async-trait", + "datafusion-common", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-plan", + "parking_lot", +] + +[[package]] +name = "datafusion-sql" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fa0d133ddf8b9b3b872acac900157f783e7b879fe9a6bccf389abebbfac45ec1" +dependencies = [ + "arrow", + "bigdecimal", + "chrono", + "datafusion-common", + "datafusion-expr", + "datafusion-functions-nested", + "indexmap", + "log", + "recursive", + "regex", + "sqlparser", +] + +[[package]] +name = "digest" +version = "0.10.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" +dependencies = [ + "block-buffer", + "crypto-common", + "subtle", +] + +[[package]] +name = "displaydoc" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "97369cbbc041bc366949bc74d34658d6cda5621039731c6310521892a3a20ae0" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "either" +version = "1.15.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "48c757948c5ede0e46177b7add2e67155f70e33c07fea8284df6576da70b3719" + +[[package]] +name = "equivalent" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" + +[[package]] +name = "errno" +version = "0.3.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" +dependencies = [ + "libc", + "windows-sys 0.61.2", +] + +[[package]] +name = "fastrand" +version = "2.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f1f227452a390804cdb637b74a86990f2a7d7ba4b7d5693aac9b4dd6defd8d6" + +[[package]] +name = "find-msvc-tools" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582" + +[[package]] +name = "fixedbitset" +version = "0.5.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d674e81391d1e1ab681a28d99df07927c6d4aa5b027d7da16ba32d1d21ecd99" + +[[package]] +name = "flatbuffers" +version = "25.12.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "35f6839d7b3b98adde531effaf34f0c2badc6f4735d26fe74709d8e513a96ef3" +dependencies = [ + "bitflags", + "rustc_version", +] + +[[package]] +name = "flate2" +version = "1.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "843fba2746e448b37e26a819579957415c8cef339bf08564fe8b7ddbd959573c" +dependencies = [ + "crc32fast", + "miniz_oxide", + "zlib-rs", +] + +[[package]] +name = "foldhash" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2" + +[[package]] +name = "foldhash" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb" + +[[package]] +name = "form_urlencoded" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb4cb245038516f5f85277875cdaa4f7d2c9a0fa0468de06ed190163b1581fcf" +dependencies = [ + "percent-encoding", +] + +[[package]] +name = "futures" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b147ee9d1f6d097cef9ce628cd2ee62288d963e16fb287bd9286455b241382d" +dependencies = [ + "futures-channel", + "futures-core", + "futures-executor", + "futures-io", + "futures-sink", + "futures-task", + "futures-util", +] + +[[package]] +name = "futures-channel" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "07bbe89c50d7a535e539b8c17bc0b49bdb77747034daa8087407d655f3f7cc1d" +dependencies = [ + "futures-core", + "futures-sink", +] + +[[package]] +name = "futures-core" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e3450815272ef58cec6d564423f6e755e25379b217b0bc688e295ba24df6b1d" + +[[package]] +name = "futures-executor" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "baf29c38818342a3b26b5b923639e7b1f4a61fc5e76102d4b1981c6dc7a7579d" +dependencies = [ + "futures-core", + "futures-task", + "futures-util", +] + +[[package]] +name = "futures-io" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cecba35d7ad927e23624b22ad55235f2239cfa44fd10428eecbeba6d6a717718" + +[[package]] +name = "futures-macro" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e835b70203e41293343137df5c0664546da5745f82ec9b84d40be8336958447b" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "futures-sink" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c39754e157331b013978ec91992bde1ac089843443c49cbc7f46150b0fad0893" + +[[package]] +name = "futures-task" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "037711b3d59c33004d3856fbdc83b99d4ff37a24768fa1be9ce3538a1cde4393" + +[[package]] +name = "futures-util" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "389ca41296e6190b48053de0321d02a77f32f8a5d2461dd38762c0593805c6d6" +dependencies = [ + "futures-channel", + "futures-core", + "futures-io", + "futures-macro", + "futures-sink", + "futures-task", + "memchr", + "pin-project-lite", + "slab", +] + +[[package]] +name = "generic-array" +version = "0.14.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" +dependencies = [ + "typenum", + "version_check", +] + +[[package]] +name = "getrandom" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0" +dependencies = [ + "cfg-if", + "libc", + "wasi", +] + +[[package]] +name = "getrandom" +version = "0.3.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd" +dependencies = [ + "cfg-if", + "libc", + "r-efi 5.3.0", + "wasip2", +] + +[[package]] +name = "getrandom" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0de51e6874e94e7bf76d726fc5d13ba782deca734ff60d5bb2fb2607c7406555" +dependencies = [ + "cfg-if", + "libc", + "r-efi 6.0.0", + "wasip2", + "wasip3", +] + +[[package]] +name = "glob" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0cc23270f6e1808e30a928bdc84dea0b9b4136a8bc82338574f23baf47bbd280" + +[[package]] +name = "half" +version = "2.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ea2d84b969582b4b1864a92dc5d27cd2b77b622a8d79306834f1be5ba20d84b" +dependencies = [ + "cfg-if", + "crunchy", + "num-traits", + "zerocopy", +] + +[[package]] +name = "hashbrown" +version = "0.14.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1" + +[[package]] +name = "hashbrown" +version = "0.15.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1" +dependencies = [ + "foldhash 0.1.5", +] + +[[package]] +name = "hashbrown" +version = "0.16.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "841d1cc9bed7f9236f321df977030373f4a4163ae1a7dbfe1a51a2c1a51d9100" +dependencies = [ + "allocator-api2", + "equivalent", + "foldhash 0.2.0", +] + +[[package]] +name = "hashbrown" +version = "0.17.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" + +[[package]] +name = "heck" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" + +[[package]] +name = "hex" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" + +[[package]] +name = "http" +version = "1.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e3ba2a386d7f85a81f119ad7498ebe444d2e22c2af0b86b069416ace48b3311a" +dependencies = [ + "bytes", + "itoa", +] + +[[package]] +name = "humantime" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "135b12329e5e3ce057a9f972339ea52bc954fe1e9358ef27f95e89716fbc5424" + +[[package]] +name = "iana-time-zone" +version = "0.1.65" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e31bc9ad994ba00e440a8aa5c9ef0ec67d5cb5e5cb0cc7f8b744a35b389cc470" +dependencies = [ + "android_system_properties", + "core-foundation-sys", + "iana-time-zone-haiku", + "js-sys", + "log", + "wasm-bindgen", + "windows-core", +] + +[[package]] +name = "iana-time-zone-haiku" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f" +dependencies = [ + "cc", +] + +[[package]] +name = "icu_collections" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2984d1cd16c883d7935b9e07e44071dca8d917fd52ecc02c04d5fa0b5a3f191c" +dependencies = [ + "displaydoc", + "potential_utf", + "utf8_iter", + "yoke", + "zerofrom", + "zerovec", +] + +[[package]] +name = "icu_locale_core" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92219b62b3e2b4d88ac5119f8904c10f8f61bf7e95b640d25ba3075e6cac2c29" +dependencies = [ + "displaydoc", + "litemap", + "tinystr", + "writeable", + "zerovec", +] + +[[package]] +name = "icu_normalizer" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c56e5ee99d6e3d33bd91c5d85458b6005a22140021cc324cea84dd0e72cff3b4" +dependencies = [ + "icu_collections", + "icu_normalizer_data", + "icu_properties", + "icu_provider", + "smallvec", + "zerovec", +] + +[[package]] +name = "icu_normalizer_data" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "da3be0ae77ea334f4da67c12f149704f19f81d1adf7c51cf482943e84a2bad38" + +[[package]] +name = "icu_properties" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bee3b67d0ea5c2cca5003417989af8996f8604e34fb9ddf96208a033901e70de" +dependencies = [ + "icu_collections", + "icu_locale_core", + "icu_properties_data", + "icu_provider", + "zerotrie", + "zerovec", +] + +[[package]] +name = "icu_properties_data" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e2bbb201e0c04f7b4b3e14382af113e17ba4f63e2c9d2ee626b720cbce54a14" + +[[package]] +name = "icu_provider" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "139c4cf31c8b5f33d7e199446eff9c1e02decfc2f0eec2c8d71f65befa45b421" +dependencies = [ + "displaydoc", + "icu_locale_core", + "writeable", + "yoke", + "zerofrom", + "zerotrie", + "zerovec", +] + +[[package]] +name = "id-arena" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3d3067d79b975e8844ca9eb072e16b31c3c1c36928edf9c6789548c524d0d954" + +[[package]] +name = "idna" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3b0875f23caa03898994f6ddc501886a45c7d3d62d04d2d90788d47be1b1e4de" +dependencies = [ + "idna_adapter", + "smallvec", + "utf8_iter", +] + +[[package]] +name = "idna_adapter" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb68373c0d6620ef8105e855e7745e18b0d00d3bdb07fb532e434244cdb9a714" +dependencies = [ + "icu_normalizer", + "icu_properties", +] + +[[package]] +name = "indexmap" +version = "2.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9" +dependencies = [ + "equivalent", + "hashbrown 0.17.1", + "serde", + "serde_core", +] + +[[package]] +name = "integer-encoding" +version = "3.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8bb03732005da905c88227371639bf1ad885cc712789c011c31c5fb3ab3ccf02" + +[[package]] +name = "itertools" +version = "0.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b192c782037fadd9cfa75548310488aabdbf3d2da73885b31bd0abd03351285" +dependencies = [ + "either", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "jni" +version = "0.21.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1a87aa2bb7d2af34197c04845522473242e1aa17c12f4935d5856491a7fb8c97" +dependencies = [ + "cesu8", + "cfg-if", + "combine", + "jni-sys 0.3.1", + "log", + "thiserror 1.0.69", + "walkdir", + "windows-sys 0.45.0", +] + +[[package]] +name = "jni-sys" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41a652e1f9b6e0275df1f15b32661cf0d4b78d4d87ddec5e0c3c20f097433258" +dependencies = [ + "jni-sys 0.4.1", +] + +[[package]] +name = "jni-sys" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c6377a88cb3910bee9b0fa88d4f42e1d2da8e79915598f65fb0c7ee14c878af2" +dependencies = [ + "jni-sys-macros", +] + +[[package]] +name = "jni-sys-macros" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "38c0b942f458fe50cdac086d2f946512305e5631e720728f2a61aabcd47a6264" +dependencies = [ + "quote", + "syn", +] + +[[package]] +name = "jobserver" +version = "0.1.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9afb3de4395d6b3e67a780b6de64b51c978ecf11cb9a462c66be7d4ca9039d33" +dependencies = [ + "getrandom 0.3.4", + "libc", +] + +[[package]] +name = "js-sys" +version = "0.3.98" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "67df7112613f8bfd9150013a0314e196f4800d3201ae742489d999db2f979f08" +dependencies = [ + "cfg-if", + "futures-util", + "once_cell", + "wasm-bindgen", +] + +[[package]] +name = "leb128fmt" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09edd9e8b54e49e587e4f6295a7d29c3ea94d469cb40ab8ca70b288248a81db2" + +[[package]] +name = "lexical-core" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7d8d125a277f807e55a77304455eb7b1cb52f2b18c143b60e766c120bd64a594" +dependencies = [ + "lexical-parse-float", + "lexical-parse-integer", + "lexical-util", + "lexical-write-float", + "lexical-write-integer", +] + +[[package]] +name = "lexical-parse-float" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "52a9f232fbd6f550bc0137dcb5f99ab674071ac2d690ac69704593cb4abbea56" +dependencies = [ + "lexical-parse-integer", + "lexical-util", +] + +[[package]] +name = "lexical-parse-integer" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a7a039f8fb9c19c996cd7b2fcce303c1b2874fe1aca544edc85c4a5f8489b34" +dependencies = [ + "lexical-util", +] + +[[package]] +name = "lexical-util" +version = "1.0.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2604dd126bb14f13fb5d1bd6a66155079cb9fa655b37f875b3a742c705dbed17" + +[[package]] +name = "lexical-write-float" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "50c438c87c013188d415fbabbb1dceb44249ab81664efbd31b14ae55dabb6361" +dependencies = [ + "lexical-util", + "lexical-write-integer", +] + +[[package]] +name = "lexical-write-integer" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "409851a618475d2d5796377cad353802345cba92c867d9fbcde9cf4eac4e14df" +dependencies = [ + "lexical-util", +] + +[[package]] +name = "libbz2-rs-sys" +version = "0.2.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b3a6a8c165077efc8f3a971534c50ea6a1a18b329ef4a66e897a7e3a1494565f" + +[[package]] +name = "libc" +version = "0.2.186" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" + +[[package]] +name = "liblzma" +version = "0.4.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6033b77c21d1f56deeae8014eb9fbe7bdf1765185a6c508b5ca82eeaed7f899" +dependencies = [ + "liblzma-sys", +] + +[[package]] +name = "liblzma-sys" +version = "0.4.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1a60851d15cd8c5346eca4ab8babff585be2ae4bc8097c067291d3ffe2add3b6" +dependencies = [ + "cc", + "libc", + "pkg-config", +] + +[[package]] +name = "libm" +version = "0.2.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981" + +[[package]] +name = "linux-raw-sys" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" + +[[package]] +name = "litemap" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92daf443525c4cce67b150400bc2316076100ce0b3686209eb8cf3c31612e6f0" + +[[package]] +name = "lock_api" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "224399e74b87b5f3557511d98dff8b14089b3dadafcab6bb93eab67d3aace965" +dependencies = [ + "scopeguard", +] + +[[package]] +name = "log" +version = "0.4.29" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5e5032e24019045c762d3c0f28f5b6b8bbf38563a65908389bf7978758920897" + +[[package]] +name = "lz4_flex" +version = "0.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7ef0d4ed8669f8f8826eb00dc878084aa8f253506c4fd5e8f58f5bce72ddb97e" +dependencies = [ + "twox-hash", +] + +[[package]] +name = "md-5" +version = "0.10.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d89e7ee0cfbedfc4da3340218492196241d89eefb6dab27de5df917a6d2e78cf" +dependencies = [ + "cfg-if", + "digest", +] + +[[package]] +name = "memchr" +version = "2.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8ca58f447f06ed17d5fc4043ce1b10dd205e060fb3ce5b979b8ed8e59ff3f79" + +[[package]] +name = "miniz_oxide" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fa76a2c86f704bdb222d66965fb3d63269ce38518b83cb0575fca855ebb6316" +dependencies = [ + "adler2", + "simd-adler32", +] + +[[package]] +name = "num-bigint" +version = "0.4.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a5e44f723f1133c9deac646763579fdb3ac745e418f2a7af9cd0c431da1f20b9" +dependencies = [ + "num-integer", + "num-traits", +] + +[[package]] +name = "num-complex" +version = "0.4.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "73f88a1307638156682bada9d7604135552957b7818057dcef22705b4d509495" +dependencies = [ + "num-traits", +] + +[[package]] +name = "num-integer" +version = "0.1.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7969661fd2958a5cb096e56c8e1ad0444ac2bbcd0061bd28660485a44879858f" +dependencies = [ + "num-traits", +] + +[[package]] +name = "num-traits" +version = "0.2.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" +dependencies = [ + "autocfg", + "libm", +] + +[[package]] +name = "object" +version = "0.37.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff76201f031d8863c38aa7f905eca4f53abbfa15f609db4277d44cd8938f33fe" +dependencies = [ + "memchr", +] + +[[package]] +name = "object_store" +version = "0.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "622acbc9100d3c10e2ee15804b0caa40e55c933d5aa53814cd520805b7958a49" +dependencies = [ + "async-trait", + "bytes", + "chrono", + "futures-channel", + "futures-core", + "futures-util", + "http", + "humantime", + "itertools", + "parking_lot", + "percent-encoding", + "thiserror 2.0.18", + "tokio", + "tracing", + "url", + "walkdir", + "wasm-bindgen-futures", + "web-time", +] + +[[package]] +name = "once_cell" +version = "1.21.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" + +[[package]] +name = "ordered-float" +version = "2.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68f19d67e5a2795c94e73e0bb1cc1a7edeb2e28efd39e2e1c9b7a40c1108b11c" +dependencies = [ + "num-traits", +] + +[[package]] +name = "parking_lot" +version = "0.12.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "93857453250e3077bd71ff98b6a65ea6621a19bb0f559a85248955ac12c45a1a" +dependencies = [ + "lock_api", + "parking_lot_core", +] + +[[package]] +name = "parking_lot_core" +version = "0.9.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2621685985a2ebf1c516881c026032ac7deafcda1a2c9b7850dc81e3dfcb64c1" +dependencies = [ + "cfg-if", + "libc", + "redox_syscall", + "smallvec", + "windows-link", +] + +[[package]] +name = "parquet" +version = "58.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "43d7efd3052f7d6ef601085559a246bc991e9a8cc77e02753737df6322ce35f1" +dependencies = [ + "ahash", + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-ipc", + "arrow-schema", + "arrow-select", + "base64", + "brotli", + "bytes", + "chrono", + "flate2", + "futures", + "half", + "hashbrown 0.17.1", + "lz4_flex", + "num-bigint", + "num-integer", + "num-traits", + "object_store", + "paste", + "seq-macro", + "simdutf8", + "snap", + "thrift", + "tokio", + "twox-hash", + "zstd", +] + +[[package]] +name = "paste" +version = "1.0.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a" + +[[package]] +name = "percent-encoding" +version = "2.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" + +[[package]] +name = "petgraph" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8701b58ea97060d5e5b155d383a69952a60943f0e6dfe30b04c287beb0b27455" +dependencies = [ + "fixedbitset", + "hashbrown 0.15.5", + "indexmap", + "serde", +] + +[[package]] +name = "phf" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "913273894cec178f401a31ec4b656318d95473527be05c0752cc41cdc32be8b7" +dependencies = [ + "phf_shared", +] + +[[package]] +name = "phf_shared" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "06005508882fb681fd97892ecff4b7fd0fee13ef1aa569f8695dae7ab9099981" +dependencies = [ + "siphasher", +] + +[[package]] +name = "pin-project-lite" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" + +[[package]] +name = "pkg-config" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e" + +[[package]] +name = "potential_utf" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0103b1cef7ec0cf76490e969665504990193874ea05c85ff9bab8b911d0a0564" +dependencies = [ + "zerovec", +] + +[[package]] +name = "ppv-lite86" +version = "0.2.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85eae3c4ed2f50dcfe72643da4befc30deadb458a9b590d720cde2f2b1e97da9" +dependencies = [ + "zerocopy", +] + +[[package]] +name = "prettyplease" +version = "0.2.37" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b" +dependencies = [ + "proc-macro2", + "syn", +] + +[[package]] +name = "proc-macro2" +version = "1.0.106" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "psm" +version = "0.1.31" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "645dbe486e346d9b5de3ef16ede18c26e6c70ad97418f4874b8b1889d6e761ea" +dependencies = [ + "ar_archive_writer", + "cc", +] + +[[package]] +name = "quote" +version = "1.0.45" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "r-efi" +version = "5.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f" + +[[package]] +name = "r-efi" +version = "6.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" + +[[package]] +name = "rand" +version = "0.9.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "44c5af06bb1b7d3216d91932aed5265164bf384dc89cd6ba05cf59a35f5f76ea" +dependencies = [ + "rand_chacha", + "rand_core", +] + +[[package]] +name = "rand_chacha" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb" +dependencies = [ + "ppv-lite86", + "rand_core", +] + +[[package]] +name = "rand_core" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "76afc826de14238e6e8c374ddcc1fa19e374fd8dd986b0d2af0d02377261d83c" +dependencies = [ + "getrandom 0.3.4", +] + +[[package]] +name = "recursive" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0786a43debb760f491b1bc0269fe5e84155353c67482b9e60d0cfb596054b43e" +dependencies = [ + "recursive-proc-macro-impl", + "stacker", +] + +[[package]] +name = "recursive-proc-macro-impl" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "76009fbe0614077fc1a2ce255e3a1881a2e3a3527097d5dc6d8212c585e7e38b" +dependencies = [ + "quote", + "syn", +] + +[[package]] +name = "redox_syscall" +version = "0.5.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed2bf2547551a7053d6fdfafda3f938979645c44812fbfcda098faae3f1a362d" +dependencies = [ + "bitflags", +] + +[[package]] +name = "regex" +version = "1.12.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e10754a14b9137dd7b1e3e5b0493cc9171fdd105e0ab477f51b72e7f3ac0e276" +dependencies = [ + "aho-corasick", + "memchr", + "regex-automata", + "regex-syntax", +] + +[[package]] +name = "regex-automata" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6e1dd4122fc1595e8162618945476892eefca7b88c52820e74af6262213cae8f" +dependencies = [ + "aho-corasick", + "memchr", + "regex-syntax", +] + +[[package]] +name = "regex-syntax" +version = "0.8.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc897dd8d9e8bd1ed8cdad82b5966c3e0ecae09fb1907d58efaa013543185d0a" + +[[package]] +name = "rustc_version" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" +dependencies = [ + "semver", +] + +[[package]] +name = "rustix" +version = "1.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190" +dependencies = [ + "bitflags", + "errno", + "libc", + "linux-raw-sys", + "windows-sys 0.61.2", +] + +[[package]] +name = "rustversion" +version = "1.0.22" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d" + +[[package]] +name = "ryu" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f" + +[[package]] +name = "same-file" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502" +dependencies = [ + "winapi-util", +] + +[[package]] +name = "scopeguard" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" + +[[package]] +name = "semver" +version = "1.0.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" + +[[package]] +name = "seq-macro" +version = "0.3.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1bc711410fbe7399f390ca1c3b60ad0f53f80e95c5eb935e52268a0e2cd49acc" + +[[package]] +name = "serde" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" +dependencies = [ + "serde_core", +] + +[[package]] +name = "serde_core" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "serde_json" +version = "1.0.149" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "83fc039473c5595ace860d8c4fafa220ff474b3fc6bfdb4293327f1a37e94d86" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "sha2" +version = "0.10.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283" +dependencies = [ + "cfg-if", + "cpufeatures 0.2.17", + "digest", +] + +[[package]] +name = "shlex" +version = "1.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0fda2ff0d084019ba4d7c6f371c95d8fd75ce3524c3cb8fb653a3023f6323e64" + +[[package]] +name = "simd-adler32" +version = "0.3.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "703d5c7ef118737c72f1af64ad2f6f8c5e1921f818cdcb97b8fe6fc69bf66214" + +[[package]] +name = "simdutf8" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e" + +[[package]] +name = "siphasher" +version = "1.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ee5873ec9cce0195efcb7a4e9507a04cd49aec9c83d0389df45b1ef7ba2e649" + +[[package]] +name = "slab" +version = "0.4.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" + +[[package]] +name = "smallvec" +version = "1.15.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "67b1b7a3b5fe4f1376887184045fcf45c69e92af734b7aaddc05fb777b6fbd03" + +[[package]] +name = "snap" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1b6b67fb9a61334225b5b790716f609cd58395f895b3fe8b328786812a40bc3b" + +[[package]] +name = "sqlparser" +version = "0.61.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dbf5ea8d4d7c808e1af1cbabebca9a2abe603bcefc22294c5b95018d53200cb7" +dependencies = [ + "log", + "recursive", + "sqlparser_derive", +] + +[[package]] +name = "sqlparser_derive" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6dd45d8fc1c79299bfbb7190e42ccbbdf6a5f52e4a6ad98d92357ea965bd289" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "stable_deref_trait" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" + +[[package]] +name = "stacker" +version = "0.1.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "640c8cdd92b6b12f5bcb1803ca3bbf5ab96e5e6b6b96b9ab77dabe9e880b3190" +dependencies = [ + "cc", + "cfg-if", + "libc", + "psm", + "windows-sys 0.61.2", +] + +[[package]] +name = "subtle" +version = "2.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" + +[[package]] +name = "syn" +version = "2.0.117" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e665b8803e7b1d2a727f4023456bbbbe74da67099c585258af0ad9c5013b9b99" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "synstructure" +version = "0.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "tempfile" +version = "3.27.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd" +dependencies = [ + "fastrand", + "getrandom 0.4.2", + "once_cell", + "rustix", + "windows-sys 0.61.2", +] + +[[package]] +name = "thiserror" +version = "1.0.69" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6aaf5339b578ea85b50e080feb250a3e8ae8cfcdff9a461c9ec2904bc923f52" +dependencies = [ + "thiserror-impl 1.0.69", +] + +[[package]] +name = "thiserror" +version = "2.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4288b5bcbc7920c07a1149a35cf9590a2aa808e0bc1eafaade0b80947865fbc4" +dependencies = [ + "thiserror-impl 2.0.18", +] + +[[package]] +name = "thiserror-impl" +version = "1.0.69" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4fee6c4efc90059e10f81e6d42c60a18f76588c3d74cb83a0b242a2b6c7504c1" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "thiserror-impl" +version = "2.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebc4ee7f67670e9b64d05fa4253e753e016c6c95ff35b89b7941d6b856dec1d5" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "thrift" +version = "0.17.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e54bc85fc7faa8bc175c4bab5b92ba8d9a3ce893d0e9f42cc455c8ab16a9e09" +dependencies = [ + "byteorder", + "integer-encoding", + "ordered-float", +] + +[[package]] +name = "tiny-keccak" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2c9d3793400a45f954c52e73d068316d76b6f4e36977e3fcebb13a2721e80237" +dependencies = [ + "crunchy", +] + +[[package]] +name = "tinystr" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c8323304221c2a851516f22236c5722a72eaa19749016521d6dff0824447d96d" +dependencies = [ + "displaydoc", + "zerovec", +] + +[[package]] +name = "tokio" +version = "1.52.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fc7f01b389ac15039e4dc9531aa973a135d7a4135281b12d7c1bc79fd57fffe" +dependencies = [ + "bytes", + "pin-project-lite", + "tokio-macros", +] + +[[package]] +name = "tokio-macros" +version = "2.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "385a6cb71ab9ab790c5fe8d67f1645e6c450a7ce006a33de03daa956cf70a496" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "tokio-stream" +version = "0.1.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32da49809aab5c3bc678af03902d4ccddea2a87d028d86392a4b1560c6906c70" +dependencies = [ + "futures-core", + "pin-project-lite", + "tokio", + "tokio-util", +] + +[[package]] +name = "tokio-util" +version = "0.7.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ae9cec805b01e8fc3fd2fe289f89149a9b66dd16786abd8b19cfa7b48cb0098" +dependencies = [ + "bytes", + "futures-core", + "futures-sink", + "pin-project-lite", + "tokio", +] + +[[package]] +name = "tracing" +version = "0.1.44" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63e71662fa4b2a2c3a26f570f037eb95bb1f85397f3cd8076caed2f026a6d100" +dependencies = [ + "pin-project-lite", + "tracing-attributes", + "tracing-core", +] + +[[package]] +name = "tracing-attributes" +version = "0.1.31" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7490cfa5ec963746568740651ac6781f701c9c5ea257c58e057f3ba8cf69e8da" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "tracing-core" +version = "0.1.36" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "db97caf9d906fbde555dd62fa95ddba9eecfd14cb388e4f491a66d74cd5fb79a" +dependencies = [ + "once_cell", +] + +[[package]] +name = "twox-hash" +version = "2.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ea3136b675547379c4bd395ca6b938e5ad3c3d20fad76e7fe85f9e0d011419c" + +[[package]] +name = "typenum" +version = "1.20.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "40ce102ab67701b8526c123c1bab5cbe42d7040ccfd0f64af1a385808d2f43de" + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "unicode-segmentation" +version = "1.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9629274872b2bfaf8d66f5f15725007f635594914870f65218920345aa11aa8c" + +[[package]] +name = "unicode-width" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b4ac048d71ede7ee76d585517add45da530660ef4390e49b098733c6e897f254" + +[[package]] +name = "unicode-xid" +version = "0.2.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853" + +[[package]] +name = "url" +version = "2.5.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff67a8a4397373c3ef660812acab3268222035010ab8680ec4215f38ba3d0eed" +dependencies = [ + "form_urlencoded", + "idna", + "percent-encoding", + "serde", +] + +[[package]] +name = "utf8_iter" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be" + +[[package]] +name = "uuid" +version = "1.23.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ddd74a9687298c6858e9b88ec8935ec45d22e8fd5e6394fa1bd4e99a87789c76" +dependencies = [ + "getrandom 0.4.2", + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "version_check" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" + +[[package]] +name = "walkdir" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b" +dependencies = [ + "same-file", + "winapi-util", +] + +[[package]] +name = "wasi" +version = "0.11.1+wasi-snapshot-preview1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" + +[[package]] +name = "wasip2" +version = "1.0.3+wasi-0.2.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "20064672db26d7cdc89c7798c48a0fdfac8213434a1186e5ef29fd560ae223d6" +dependencies = [ + "wit-bindgen 0.57.1", +] + +[[package]] +name = "wasip3" +version = "0.4.0+wasi-0.3.0-rc-2026-01-06" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5428f8bf88ea5ddc08faddef2ac4a67e390b88186c703ce6dbd955e1c145aca5" +dependencies = [ + "wit-bindgen 0.51.0", +] + +[[package]] +name = "wasm-bindgen" +version = "0.2.121" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "49ace1d07c165b0864824eee619580c4689389afa9dc9ed3a4c75040d82e6790" +dependencies = [ + "cfg-if", + "once_cell", + "rustversion", + "wasm-bindgen-macro", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-futures" +version = "0.4.71" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "96492d0d3ffba25305a7dc88720d250b1401d7edca02cc3bcd50633b424673b8" +dependencies = [ + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "wasm-bindgen-macro" +version = "0.2.121" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e68e6f4afd367a562002c05637acb8578ff2dea1943df76afb9e83d177c8578" +dependencies = [ + "quote", + "wasm-bindgen-macro-support", +] + +[[package]] +name = "wasm-bindgen-macro-support" +version = "0.2.121" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d95a9ec35c64b2a7cb35d3fead40c4238d0940c86d107136999567a4703259f2" +dependencies = [ + "bumpalo", + "proc-macro2", + "quote", + "syn", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-shared" +version = "0.2.121" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c4e0100b01e9f0d03189a92b96772a1fb998639d981193d7dbab487302513441" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "wasm-encoder" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "990065f2fe63003fe337b932cfb5e3b80e0b4d0f5ff650e6985b1048f62c8319" +dependencies = [ + "leb128fmt", + "wasmparser", +] + +[[package]] +name = "wasm-metadata" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bb0e353e6a2fbdc176932bbaab493762eb1255a7900fe0fea1a2f96c296cc909" +dependencies = [ + "anyhow", + "indexmap", + "wasm-encoder", + "wasmparser", +] + +[[package]] +name = "wasmparser" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47b807c72e1bac69382b3a6fb3dbe8ea4c0ed87ff5629b8685ae6b9a611028fe" +dependencies = [ + "bitflags", + "hashbrown 0.15.5", + "indexmap", + "semver", +] + +[[package]] +name = "web-time" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5a6580f308b1fad9207618087a65c04e7a10bc77e02c8e84e9b00dd4b12fa0bb" +dependencies = [ + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "winapi-util" +version = "0.1.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" +dependencies = [ + "windows-sys 0.61.2", +] + +[[package]] +name = "windows-core" +version = "0.62.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb" +dependencies = [ + "windows-implement", + "windows-interface", + "windows-link", + "windows-result", + "windows-strings", +] + +[[package]] +name = "windows-implement" +version = "0.60.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "windows-interface" +version = "0.59.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-result" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5" +dependencies = [ + "windows-link", +] + +[[package]] +name = "windows-strings" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091" +dependencies = [ + "windows-link", +] + +[[package]] +name = "windows-sys" +version = "0.45.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "75283be5efb2831d37ea142365f009c02ec203cd29a3ebecbc093d52315b66d0" +dependencies = [ + "windows-targets", +] + +[[package]] +name = "windows-sys" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" +dependencies = [ + "windows-link", +] + +[[package]] +name = "windows-targets" +version = "0.42.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e5180c00cd44c9b1c88adb3693291f1cd93605ded80c250a75d472756b4d071" +dependencies = [ + "windows_aarch64_gnullvm", + "windows_aarch64_msvc", + "windows_i686_gnu", + "windows_i686_msvc", + "windows_x86_64_gnu", + "windows_x86_64_gnullvm", + "windows_x86_64_msvc", +] + +[[package]] +name = "windows_aarch64_gnullvm" +version = "0.42.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "597a5118570b68bc08d8d59125332c54f1ba9d9adeedeef5b99b02ba2b0698f8" + +[[package]] +name = "windows_aarch64_msvc" +version = "0.42.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e08e8864a60f06ef0d0ff4ba04124db8b0fb3be5776a5cd47641e942e58c4d43" + +[[package]] +name = "windows_i686_gnu" +version = "0.42.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c61d927d8da41da96a81f029489353e68739737d3beca43145c8afec9a31a84f" + +[[package]] +name = "windows_i686_msvc" +version = "0.42.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "44d840b6ec649f480a41c8d80f9c65108b92d89345dd94027bfe06ac444d1060" + +[[package]] +name = "windows_x86_64_gnu" +version = "0.42.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8de912b8b8feb55c064867cf047dda097f92d51efad5b491dfb98f6bbb70cb36" + +[[package]] +name = "windows_x86_64_gnullvm" +version = "0.42.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "26d41b46a36d453748aedef1486d5c7a85db22e56aff34643984ea85514e94a3" + +[[package]] +name = "windows_x86_64_msvc" +version = "0.42.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9aec5da331524158c6d1a4ac0ab1541149c0b9505fde06423b02f5ef0106b9f0" + +[[package]] +name = "wit-bindgen" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d7249219f66ced02969388cf2bb044a09756a083d0fab1e566056b04d9fbcaa5" +dependencies = [ + "wit-bindgen-rust-macro", +] + +[[package]] +name = "wit-bindgen" +version = "0.57.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ebf944e87a7c253233ad6766e082e3cd714b5d03812acc24c318f549614536e" + +[[package]] +name = "wit-bindgen-core" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ea61de684c3ea68cb082b7a88508a8b27fcc8b797d738bfc99a82facf1d752dc" +dependencies = [ + "anyhow", + "heck", + "wit-parser", +] + +[[package]] +name = "wit-bindgen-rust" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7c566e0f4b284dd6561c786d9cb0142da491f46a9fbed79ea69cdad5db17f21" +dependencies = [ + "anyhow", + "heck", + "indexmap", + "prettyplease", + "syn", + "wasm-metadata", + "wit-bindgen-core", + "wit-component", +] + +[[package]] +name = "wit-bindgen-rust-macro" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c0f9bfd77e6a48eccf51359e3ae77140a7f50b1e2ebfe62422d8afdaffab17a" +dependencies = [ + "anyhow", + "prettyplease", + "proc-macro2", + "quote", + "syn", + "wit-bindgen-core", + "wit-bindgen-rust", +] + +[[package]] +name = "wit-component" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9d66ea20e9553b30172b5e831994e35fbde2d165325bec84fc43dbf6f4eb9cb2" +dependencies = [ + "anyhow", + "bitflags", + "indexmap", + "log", + "serde", + "serde_derive", + "serde_json", + "wasm-encoder", + "wasm-metadata", + "wasmparser", + "wit-parser", +] + +[[package]] +name = "wit-parser" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ecc8ac4bc1dc3381b7f59c34f00b67e18f910c2c0f50015669dde7def656a736" +dependencies = [ + "anyhow", + "id-arena", + "indexmap", + "log", + "semver", + "serde", + "serde_derive", + "serde_json", + "unicode-xid", + "wasmparser", +] + +[[package]] +name = "writeable" +version = "0.6.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4" + +[[package]] +name = "yoke" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "abe8c5fda708d9ca3df187cae8bfb9ceda00dd96231bed36e445a1a48e66f9ca" +dependencies = [ + "stable_deref_trait", + "yoke-derive", + "zerofrom", +] + +[[package]] +name = "yoke-derive" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "de844c262c8848816172cef550288e7dc6c7b7814b4ee56b3e1553f275f1858e" +dependencies = [ + "proc-macro2", + "quote", + "syn", + "synstructure", +] + +[[package]] +name = "zerocopy" +version = "0.8.48" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "eed437bf9d6692032087e337407a86f04cd8d6a16a37199ed57949d415bd68e9" +dependencies = [ + "zerocopy-derive", +] + +[[package]] +name = "zerocopy-derive" +version = "0.8.48" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "70e3cd084b1788766f53af483dd21f93881ff30d7320490ec3ef7526d203bad4" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "zerofrom" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69faa1f2a1ea75661980b013019ed6687ed0e83d069bc1114e2cc74c6c04c4df" +dependencies = [ + "zerofrom-derive", +] + +[[package]] +name = "zerofrom-derive" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "11532158c46691caf0f2593ea8358fed6bbf68a0315e80aae9bd41fbade684a1" +dependencies = [ + "proc-macro2", + "quote", + "syn", + "synstructure", +] + +[[package]] +name = "zerotrie" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0f9152d31db0792fa83f70fb2f83148effb5c1f5b8c7686c3459e361d9bc20bf" +dependencies = [ + "displaydoc", + "yoke", + "zerofrom", +] + +[[package]] +name = "zerovec" +version = "0.11.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "90f911cbc359ab6af17377d242225f4d75119aec87ea711a880987b18cd7b239" +dependencies = [ + "yoke", + "zerofrom", + "zerovec-derive", +] + +[[package]] +name = "zerovec-derive" +version = "0.11.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "625dc425cab0dca6dc3c3319506e6593dcb08a9f387ea3b284dbd52a92c40555" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "zlib-rs" +version = "0.6.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3be3d40e40a133f9c916ee3f9f4fa2d9d63435b5fbe1bfc6d9dae0aa0ada1513" + +[[package]] +name = "zmij" +version = "1.0.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa" + +[[package]] +name = "zstd" +version = "0.13.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e91ee311a569c327171651566e07972200e76fcfe2242a4fa446149a3881c08a" +dependencies = [ + "zstd-safe", +] + +[[package]] +name = "zstd-safe" +version = "7.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f49c4d5f0abb602a93fb8736af2a4f4dd9512e36f7f570d66e65ff867ed3b9d" +dependencies = [ + "zstd-sys", +] + +[[package]] +name = "zstd-sys" +version = "2.0.16+zstd.1.5.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91e19ebc2adc8f83e43039e79776e3fda8ca919132d68a1fed6a5faca2683748" +dependencies = [ + "cc", + "pkg-config", +] diff --git a/native/Cargo.toml b/native/Cargo.toml new file mode 100644 index 0000000..084090e --- /dev/null +++ b/native/Cargo.toml @@ -0,0 +1,31 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +[package] +name = "datafusion-jni" +version = "0.1.0" +edition = "2021" +publish = false + +[lib] +crate-type = ["cdylib"] + +[dependencies] +arrow = { version = "58", features = ["ffi"] } +datafusion = "53" +jni = "0.21" +tokio = { version = "1", features = ["rt-multi-thread"] } diff --git a/native/src/errors.rs b/native/src/errors.rs new file mode 100644 index 0000000..e779bb7 --- /dev/null +++ b/native/src/errors.rs @@ -0,0 +1,58 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use std::any::Any; +use std::error::Error; +use std::panic::{catch_unwind, AssertUnwindSafe}; + +use jni::JNIEnv; + +pub type JniResult = Result>; + +pub fn try_unwrap_or_throw(env: &mut JNIEnv, default: T, f: F) -> T +where + F: FnOnce(&mut JNIEnv) -> JniResult, +{ + match catch_unwind(AssertUnwindSafe(|| f(env))) { + Ok(Ok(value)) => value, + Ok(Err(err)) => { + throw_runtime_exception(env, &err.to_string()); + default + } + Err(panic) => { + throw_runtime_exception(env, &panic_message(&panic)); + default + } + } +} + +fn throw_runtime_exception(env: &mut JNIEnv, message: &str) { + if env.exception_check().unwrap_or(false) { + return; + } + let _ = env.throw_new("java/lang/RuntimeException", message); +} + +fn panic_message(panic: &Box) -> String { + if let Some(s) = panic.downcast_ref::() { + s.clone() + } else if let Some(s) = panic.downcast_ref::<&str>() { + (*s).to_string() + } else { + "rust panic with non-string payload".to_string() + } +} diff --git a/native/src/lib.rs b/native/src/lib.rs new file mode 100644 index 0000000..947d47f --- /dev/null +++ b/native/src/lib.rs @@ -0,0 +1,154 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +mod errors; + +use std::sync::{Arc, OnceLock}; + +use datafusion::arrow::datatypes::SchemaRef; +use datafusion::arrow::ffi_stream::FFI_ArrowArrayStream; +use datafusion::arrow::record_batch::RecordBatchIterator; +use datafusion::dataframe::DataFrame; +use datafusion::error::DataFusionError; +use datafusion::prelude::{ParquetReadOptions, SessionContext}; +use jni::objects::{JClass, JString}; +use jni::sys::jlong; +use jni::JNIEnv; +use tokio::runtime::Runtime; + +use crate::errors::{try_unwrap_or_throw, JniResult}; + +fn runtime() -> &'static Runtime { + static RT: OnceLock = OnceLock::new(); + RT.get_or_init(|| Runtime::new().expect("failed to create Tokio runtime")) +} + +#[no_mangle] +pub extern "system" fn Java_org_apache_datafusion_SessionContext_createSessionContext<'local>( + mut env: JNIEnv<'local>, + _class: JClass<'local>, +) -> jlong { + try_unwrap_or_throw(&mut env, 0, |_env| -> JniResult { + let ctx = SessionContext::new(); + Ok(Box::into_raw(Box::new(ctx)) as jlong) + }) +} + +#[no_mangle] +pub extern "system" fn Java_org_apache_datafusion_SessionContext_createDataFrame<'local>( + mut env: JNIEnv<'local>, + _class: JClass<'local>, + handle: jlong, + sql: JString<'local>, +) -> jlong { + try_unwrap_or_throw(&mut env, 0, |env| -> JniResult { + if handle == 0 { + return Err("SessionContext handle is null".into()); + } + let ctx = unsafe { &*(handle as *const SessionContext) }; + let sql_str: String = env.get_string(&sql)?.into(); + + let df = runtime().block_on(async { ctx.sql(&sql_str).await })?; + Ok(Box::into_raw(Box::new(df)) as jlong) + }) +} + +#[no_mangle] +pub extern "system" fn Java_org_apache_datafusion_DataFrame_collectDataFrame<'local>( + mut env: JNIEnv<'local>, + _class: JClass<'local>, + handle: jlong, + ffi_stream_addr: jlong, +) { + try_unwrap_or_throw(&mut env, (), |_env| -> JniResult<()> { + if handle == 0 { + return Err("DataFrame handle is null".into()); + } + if ffi_stream_addr == 0 { + return Err("ffi stream address is null".into()); + } + let df = unsafe { *Box::from_raw(handle as *mut DataFrame) }; + + let ffi: FFI_ArrowArrayStream = runtime().block_on(async { + let schema: SchemaRef = Arc::new(df.schema().as_arrow().clone()); + let batches = df.collect().await?; + let iter = RecordBatchIterator::new(batches.into_iter().map(Ok), schema); + Ok::<_, DataFusionError>(FFI_ArrowArrayStream::new(Box::new(iter))) + })?; + + unsafe { + std::ptr::write(ffi_stream_addr as *mut FFI_ArrowArrayStream, ffi); + } + Ok(()) + }) +} + +#[no_mangle] +pub extern "system" fn Java_org_apache_datafusion_DataFrame_closeDataFrame<'local>( + mut env: JNIEnv<'local>, + _class: JClass<'local>, + handle: jlong, +) { + try_unwrap_or_throw(&mut env, (), |_env| -> JniResult<()> { + if handle != 0 { + unsafe { + drop(Box::from_raw(handle as *mut DataFrame)); + } + } + Ok(()) + }) +} + +#[no_mangle] +pub extern "system" fn Java_org_apache_datafusion_SessionContext_closeSessionContext<'local>( + mut env: JNIEnv<'local>, + _class: JClass<'local>, + handle: jlong, +) { + try_unwrap_or_throw(&mut env, (), |_env| -> JniResult<()> { + if handle != 0 { + unsafe { + drop(Box::from_raw(handle as *mut SessionContext)); + } + } + Ok(()) + }) +} + +#[no_mangle] +pub extern "system" fn Java_org_apache_datafusion_SessionContext_registerParquet<'local>( + mut env: JNIEnv<'local>, + _class: JClass<'local>, + handle: jlong, + name: JString<'local>, + path: JString<'local>, +) { + try_unwrap_or_throw(&mut env, (), |env| -> JniResult<()> { + if handle == 0 { + return Err("SessionContext handle is null".into()); + } + let ctx = unsafe { &*(handle as *const SessionContext) }; + let name: String = env.get_string(&name)?.into(); + let path: String = env.get_string(&path)?.into(); + runtime().block_on(async { + ctx.register_parquet(&name, &path, ParquetReadOptions::default()) + .await?; + Ok::<(), DataFusionError>(()) + })?; + Ok(()) + }) +} diff --git a/pom.xml b/pom.xml new file mode 100644 index 0000000..d2b1b91 --- /dev/null +++ b/pom.xml @@ -0,0 +1,143 @@ + + + + 4.0.0 + + org.apache.datafusion + datafusion-java + 0.1.0-SNAPSHOT + jar + + + 17 + 17 + UTF-8 + 5.11.3 + + + + + org.junit.jupiter + junit-jupiter + ${junit.version} + test + + + org.apache.arrow + arrow-vector + 19.0.0 + + + org.apache.arrow + arrow-c-data + 19.0.0 + + + org.apache.arrow + arrow-memory-netty + 19.0.0 + runtime + + + + + + + org.apache.maven.plugins + maven-surefire-plugin + 3.5.2 + + -Djava.library.path=${project.basedir}/native/target/debug --add-opens=java.base/java.nio=ALL-UNNAMED + + + + com.diffplug.spotless + spotless-maven-plugin + 2.44.0 + + + + 1.25.2 + + true + + + + java,javax,org,com, + + + + + + + + verify + + check + + + + + + org.apache.rat + apache-rat-plugin + 0.16.1 + + + verify + + check + + + + + true + + + LICENSE.txt + NOTICE.txt + + README.md + docs/** + + .gitignore + .idea/** + .vscode/** + **/*.iml + **/.DS_Store + + mvnw + mvnw.cmd + .mvn/** + + target/** + native/target/** + tpch-data/** + + native/Cargo.lock + + + + + + diff --git a/src/main/java/org/apache/datafusion/DataFrame.java b/src/main/java/org/apache/datafusion/DataFrame.java new file mode 100644 index 0000000..f285879 --- /dev/null +++ b/src/main/java/org/apache/datafusion/DataFrame.java @@ -0,0 +1,84 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.datafusion; + +import org.apache.arrow.c.ArrowArrayStream; +import org.apache.arrow.c.Data; +import org.apache.arrow.memory.BufferAllocator; +import org.apache.arrow.vector.ipc.ArrowReader; + +/** + * A lazy representation of a query plan, mirroring the Rust DataFusion {@code DataFrame}. Created + * by {@link SessionContext#sql(String)} or other planning entry points and executed by {@link + * #collect}. + * + *

Instances are not thread-safe and must be closed. {@link #collect} consumes + * the DataFrame: a successfully collected DataFrame cannot be collected again, and {@link #close()} + * on an already-collected instance is a no-op. + */ +public final class DataFrame implements AutoCloseable { + static { + NativeLibraryLoader.loadLibrary(); + } + + private long nativeHandle; + + DataFrame(long nativeHandle) { + if (nativeHandle == 0) { + throw new IllegalArgumentException("DataFrame native handle is null"); + } + this.nativeHandle = nativeHandle; + } + + /** + * Execute the plan and return its record batches as an {@link ArrowReader}. + * + *

Consumes this DataFrame: the native plan is released as soon as the stream is established. + * The caller is responsible for closing the returned reader, and the supplied allocator must + * outlive it. + */ + public ArrowReader collect(BufferAllocator allocator) { + if (nativeHandle == 0) { + throw new IllegalStateException("DataFrame is closed or already collected"); + } + ArrowArrayStream stream = ArrowArrayStream.allocateNew(allocator); + long handle = nativeHandle; + nativeHandle = 0; + try { + collectDataFrame(handle, stream.memoryAddress()); + return Data.importArrayStream(allocator, stream); + } catch (Throwable e) { + stream.close(); + throw e; + } + } + + @Override + public void close() { + if (nativeHandle != 0) { + closeDataFrame(nativeHandle); + nativeHandle = 0; + } + } + + private static native void collectDataFrame(long handle, long ffiStreamAddr); + + private static native void closeDataFrame(long handle); +} diff --git a/src/main/java/org/apache/datafusion/NativeLibraryLoader.java b/src/main/java/org/apache/datafusion/NativeLibraryLoader.java new file mode 100644 index 0000000..772f726 --- /dev/null +++ b/src/main/java/org/apache/datafusion/NativeLibraryLoader.java @@ -0,0 +1,33 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.datafusion; + +public final class NativeLibraryLoader { + private static boolean loaded = false; + + private NativeLibraryLoader() {} + + public static synchronized void loadLibrary() { + if (!loaded) { + System.loadLibrary("datafusion_jni"); + loaded = true; + } + } +} diff --git a/src/main/java/org/apache/datafusion/SessionContext.java b/src/main/java/org/apache/datafusion/SessionContext.java new file mode 100644 index 0000000..ac1a45e --- /dev/null +++ b/src/main/java/org/apache/datafusion/SessionContext.java @@ -0,0 +1,78 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.datafusion; + +/** + * A DataFusion session context. + * + *

Instances are not thread-safe. Concurrent calls to any of {@link #sql}, + * {@link #registerParquet}, or {@link #close} from different threads can produce a use-after-free + * on the native side. Callers must externally synchronize, or confine each context to a single + * thread. + */ +public final class SessionContext implements AutoCloseable { + static { + NativeLibraryLoader.loadLibrary(); + } + + private long nativeHandle; + + public SessionContext() { + this.nativeHandle = createSessionContext(); + if (this.nativeHandle == 0) { + throw new RuntimeException("Failed to create native SessionContext"); + } + } + + /** + * Parse and plan {@code query}, returning a lazy {@link DataFrame}. The query is not executed + * until {@link DataFrame#collect} is called. + */ + public DataFrame sql(String query) { + if (nativeHandle == 0) { + throw new IllegalStateException("SessionContext is closed"); + } + long dfHandle = createDataFrame(nativeHandle, query); + return new DataFrame(dfHandle); + } + + public void registerParquet(String name, String path) { + if (nativeHandle == 0) { + throw new IllegalStateException("SessionContext is closed"); + } + registerParquet(nativeHandle, name, path); + } + + @Override + public void close() { + if (nativeHandle != 0) { + closeSessionContext(nativeHandle); + nativeHandle = 0; + } + } + + private static native long createSessionContext(); + + private static native long createDataFrame(long handle, String sql); + + private static native void registerParquet(long handle, String name, String path); + + private static native void closeSessionContext(long handle); +} diff --git a/src/test/java/org/apache/datafusion/SessionContextTest.java b/src/test/java/org/apache/datafusion/SessionContextTest.java new file mode 100644 index 0000000..4465546 --- /dev/null +++ b/src/test/java/org/apache/datafusion/SessionContextTest.java @@ -0,0 +1,93 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.datafusion; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.nio.file.Files; +import java.nio.file.Path; + +import org.apache.arrow.memory.BufferAllocator; +import org.apache.arrow.memory.RootAllocator; +import org.apache.arrow.vector.BigIntVector; +import org.apache.arrow.vector.VectorSchemaRoot; +import org.apache.arrow.vector.ipc.ArrowReader; +import org.junit.jupiter.api.Assumptions; +import org.junit.jupiter.api.Test; + +class SessionContextTest { + @Test + void canExecuteSelect1() throws Exception { + try (BufferAllocator allocator = new RootAllocator(); + SessionContext ctx = new SessionContext(); + DataFrame df = ctx.sql("SELECT 1"); + ArrowReader reader = df.collect(allocator)) { + assertTrue(reader.loadNextBatch()); + assertEquals(1, reader.getVectorSchemaRoot().getRowCount()); + } + } + + @Test + void selectCountStarLineitem() throws Exception { + Path lineitem = Path.of("tpch-data/sf1/lineitem.parquet"); + Assumptions.assumeTrue( + Files.exists(lineitem), "TPC-H SF1 data not found; run `make tpch-data` first"); + + try (BufferAllocator allocator = new RootAllocator(); + SessionContext ctx = new SessionContext()) { + ctx.registerParquet("lineitem", lineitem.toAbsolutePath().toString()); + try (DataFrame df = ctx.sql("SELECT COUNT(*) FROM lineitem"); + ArrowReader reader = df.collect(allocator)) { + assertTrue(reader.loadNextBatch()); + VectorSchemaRoot root = reader.getVectorSchemaRoot(); + assertEquals(1, root.getRowCount()); + BigIntVector count = (BigIntVector) root.getVector(0); + assertEquals(6_001_215L, count.get(0)); + } + } + } + + @Test + void dataFrameCloseWithoutCollectReleasesNativeHandle() { + try (SessionContext ctx = new SessionContext()) { + // Planning succeeds but we never execute. close() must release the native plan. + DataFrame df = ctx.sql("SELECT 1"); + df.close(); + // Double-close must be a no-op. + df.close(); + } + } + + @Test + void dataFrameCollectTwiceFails() { + try (BufferAllocator allocator = new RootAllocator(); + SessionContext ctx = new SessionContext(); + DataFrame df = ctx.sql("SELECT 1")) { + try (ArrowReader reader = df.collect(allocator)) { + assertTrue(reader.loadNextBatch()); + } catch (Exception e) { + throw new RuntimeException(e); + } + assertThrows(IllegalStateException.class, () -> df.collect(allocator)); + } + } +} From 969d32e14c9ae9361f8778d8ab35c9faa35db297 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Tue, 12 May 2026 17:38:02 -0600 Subject: [PATCH 02/14] chore: add .asf.yaml configuring repo metadata and branch protection --- .asf.yaml | 61 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 61 insertions(+) create mode 100644 .asf.yaml diff --git a/.asf.yaml b/.asf.yaml new file mode 100644 index 0000000..75a2b87 --- /dev/null +++ b/.asf.yaml @@ -0,0 +1,61 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# This file controls the settings of this repository +# +# See more details at +# https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features + +notifications: + commits: commits@datafusion.apache.org + issues: github@datafusion.apache.org + pullrequests: github@datafusion.apache.org + discussions: github@datafusion.apache.org + jira_options: link label worklog + +github: + description: "Java bindings for Apache DataFusion" + homepage: https://datafusion.apache.org/ + labels: + - apache + - arrow + - datafusion + - java + - jni + - jvm + - query-engine + - sql + enabled_merge_buttons: + squash: true + squash_commit_message: PR_TITLE_AND_DESC + merge: false + rebase: false + features: + issues: true + discussions: true + protected_branches: + main: + required_pull_request_reviews: + required_approving_review_count: 1 + required_status_checks: + contexts: + - "Java (spotless)" + - "Rust (cargo fmt)" + pull_requests: + allow_update_branch: true + allow_auto_merge: true + del_branch_on_merge: true From 7943e22c5acff76cc2602b72471f43a44d9d01ef Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Tue, 12 May 2026 18:14:55 -0600 Subject: [PATCH 03/14] Update native/Cargo.toml Co-authored-by: Oleks V --- native/Cargo.toml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/native/Cargo.toml b/native/Cargo.toml index 084090e..6178426 100644 --- a/native/Cargo.toml +++ b/native/Cargo.toml @@ -25,7 +25,6 @@ publish = false crate-type = ["cdylib"] [dependencies] -arrow = { version = "58", features = ["ffi"] } -datafusion = "53" +datafusion = { version = "53.1.0", default-features = false } jni = "0.21" tokio = { version = "1", features = ["rt-multi-thread"] } From e79dc7e4703ccb3877298d5cca4e76d683b4680a Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Tue, 12 May 2026 18:18:58 -0600 Subject: [PATCH 04/14] fix(native): restore datafusion default features and arrow ffi Re-enable datafusion's default features (parquet, sql) and add arrow dependency with the ffi feature so FFI_ArrowArrayStream, ctx.sql, and register_parquet compile again. --- native/Cargo.toml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/native/Cargo.toml b/native/Cargo.toml index 6178426..033fb4c 100644 --- a/native/Cargo.toml +++ b/native/Cargo.toml @@ -25,6 +25,7 @@ publish = false crate-type = ["cdylib"] [dependencies] -datafusion = { version = "53.1.0", default-features = false } +arrow = { version = "58", features = ["ffi"] } +datafusion = "53.1.0" jni = "0.21" tokio = { version = "1", features = ["rt-multi-thread"] } From 4dab96887f43ae04b69053f1363c1dcca27f93d1 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Tue, 12 May 2026 18:20:41 -0600 Subject: [PATCH 05/14] refactor: extract native library name into constant --- src/main/java/org/apache/datafusion/NativeLibraryLoader.java | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/main/java/org/apache/datafusion/NativeLibraryLoader.java b/src/main/java/org/apache/datafusion/NativeLibraryLoader.java index 772f726..2c74341 100644 --- a/src/main/java/org/apache/datafusion/NativeLibraryLoader.java +++ b/src/main/java/org/apache/datafusion/NativeLibraryLoader.java @@ -20,13 +20,14 @@ package org.apache.datafusion; public final class NativeLibraryLoader { + private static final String LIBRARY_NAME = "datafusion_jni"; private static boolean loaded = false; private NativeLibraryLoader() {} public static synchronized void loadLibrary() { if (!loaded) { - System.loadLibrary("datafusion_jni"); + System.loadLibrary(LIBRARY_NAME); loaded = true; } } From 80eaf811bb0a36a27d57a022842b408728e0fa5c Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Tue, 12 May 2026 19:58:07 -0600 Subject: [PATCH 06/14] chore: remove protected_branches block and format workflow (#7) ## Summary - Removes the entire `protected_branches` block from `.asf.yaml`. - Removes `.github/workflows/format.yml`. ## Rationale GitHub Actions has not yet been enabled on the apache repo, so the `Format` workflow ends in `startup_failure` on every run and never reports the `Java (spotless)` / `Rust (cargo fmt)` status checks. Combined with `protected_branches.main.required_status_checks` requiring those contexts, PRs are unmergeable. This change clears both sides of the deadlock: the protection block is removed so future PRs aren't blocked, and the workflow is removed so we're not carrying a check definition that can't run. Branch protection and CI can be reintroduced once Actions is enabled by INFRA and the project's review/CI practices have stabilized. --- .asf.yaml | 8 ------ .github/workflows/format.yml | 50 ------------------------------------ 2 files changed, 58 deletions(-) delete mode 100644 .github/workflows/format.yml diff --git a/.asf.yaml b/.asf.yaml index 75a2b87..70844d3 100644 --- a/.asf.yaml +++ b/.asf.yaml @@ -47,14 +47,6 @@ github: features: issues: true discussions: true - protected_branches: - main: - required_pull_request_reviews: - required_approving_review_count: 1 - required_status_checks: - contexts: - - "Java (spotless)" - - "Rust (cargo fmt)" pull_requests: allow_update_branch: true allow_auto_merge: true diff --git a/.github/workflows/format.yml b/.github/workflows/format.yml deleted file mode 100644 index 6295c32..0000000 --- a/.github/workflows/format.yml +++ /dev/null @@ -1,50 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -name: Format - -on: - push: - branches: [main] - pull_request: - branches: [main] - -jobs: - java: - name: Java (spotless) - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - uses: actions/setup-java@v4 - with: - distribution: temurin - java-version: '17' - cache: maven - - name: Check Java formatting - run: ./mvnw -B -ntp spotless:check - - rust: - name: Rust (cargo fmt) - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - uses: dtolnay/rust-toolchain@stable - with: - components: rustfmt - - name: Check Rust formatting - working-directory: native - run: cargo fmt --all -- --check From e412ab54e1ff8c1d9000afcb7c4d07b2b10ca1a7 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Tue, 12 May 2026 20:12:12 -0600 Subject: [PATCH 07/14] chore(.asf.yaml): null out main branch protection to clear stale rules (#11) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary - Sets `github.protected_branches.main: ~` in `.asf.yaml`. ## Rationale The previous PR (#7) removed the `protected_branches` block entirely, but ASF INFRA only applies settings that are present in `.asf.yaml` — it does not clear settings when keys are simply omitted. The required status checks (`Java (spotless)`, `Rust (cargo fmt)`) therefore remain configured on `main` and continue to block every PR, because the workflow that would produce those check contexts is also gone. Explicitly setting `protected_branches.main` to YAML null (`~`) tells INFRA to clear the existing branch protection rules on `main`. Branch protection can be reintroduced later once CI is in place. --- .asf.yaml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.asf.yaml b/.asf.yaml index 70844d3..cf1630f 100644 --- a/.asf.yaml +++ b/.asf.yaml @@ -47,6 +47,8 @@ github: features: issues: true discussions: true + protected_branches: + main: ~ pull_requests: allow_update_branch: true allow_auto_merge: true From 5292e05c924a2b4125f2db66898e236dc969912b Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Tue, 12 May 2026 20:16:52 -0600 Subject: [PATCH 08/14] feat(proto): generate datafusion-proto Java classes at build time (#9) --- CONTRIBUTING.md | 92 +++++++++++++++++++ pom.xml | 58 ++++++++++++ .../datafusion/proto/ProtoGenerationTest.java | 41 +++++++++ 3 files changed, 191 insertions(+) create mode 100644 CONTRIBUTING.md create mode 100644 src/test/java/org/apache/datafusion/proto/ProtoGenerationTest.java diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..302b6b3 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,92 @@ + + +# Contributing to Apache DataFusion Java + +Bug reports, design discussion, and patches are welcome. This project follows +the Apache DataFusion contribution model. + +## Filing issues and discussing changes + +- File bugs and feature requests on [GitHub issues](https://github.com/apache/datafusion-java/issues). +- For larger or design-level discussion, the mailing list is + [dev@datafusion.apache.org](mailto:dev@datafusion.apache.org). +- Please open an issue before sending a PR for any significant change so the + approach can be agreed on first. + +## Development workflow + +Branch from `main`, write changes with [conventional commit](https://www.conventionalcommits.org/) +messages in the imperative mood (e.g. `feat: add foo`, `fix(native): handle bar`), +and open a pull request targeting `main`. + +The first build in a fresh checkout reaches out to `raw.githubusercontent.com` +to fetch the DataFusion protobuf schemas (see *Updating the DataFusion / +protobuf schema version* below). Subsequent builds are offline — the +`download-maven-plugin` cache under `~/.m2/repository/.cache/` satisfies them. + +## Code style + +- Java: run `./mvnw spotless:apply` before committing. CI fails the build if + formatting drifts. +- Rust: run `cargo fmt` and `cargo clippy --all-targets -- -D warnings` inside + `native/`. +- New source files need the Apache 2.0 license header. Apache RAT enforces this + during `verify`. + +## Updating the DataFusion / protobuf schema version + +Three things must move together when bumping DataFusion: + +1. `native/Cargo.toml` — the `datafusion` crate dependency. +2. `pom.xml` — the `` Maven property. **Must equal the + Cargo version**; a mismatch means JVM-built protobuf plans won't deserialize + on the native side. +3. `pom.xml` — the `` checksums on the two `download-maven-plugin` + executions. These pin the downloaded `.proto` files; the build fails if + upstream silently re-tags them, which is the desired behavior. + +Recipe: + +```sh +# 1. Bump the Cargo dep +$EDITOR native/Cargo.toml # set datafusion = "" +(cd native && cargo update -p datafusion) + +# 2. Bump the Maven property to match +$EDITOR pom.xml # set + +# 3. Compute the new SHA-512 hashes for both `.proto` files from the upstream +# tag you just set in step 2, then paste them into the two elements +# in pom.xml. +NEW=$(grep -m1 -oE '[^<]+' pom.xml | cut -d'>' -f2) +curl -sL "https://raw.githubusercontent.com/apache/datafusion/$NEW/datafusion/proto-common/proto/datafusion_common.proto" | shasum -a 512 | awk '{print $1}' +curl -sL "https://raw.githubusercontent.com/apache/datafusion/$NEW/datafusion/proto/proto/datafusion.proto" | shasum -a 512 | awk '{print $1}' +$EDITOR pom.xml # paste the two hashes into the elements + +# Drop the local download cache so the next build re-downloads against the new hashes. +rm -rf ~/.m2/repository/.cache/download-maven-plugin target/proto + +# 4. Verify +make && make test +``` + +The protobuf runtime version (`` in `pom.xml`) tracks the +Java ecosystem (security and JDK compatibility), not DataFusion. Bump it +independently when there is a reason. diff --git a/pom.xml b/pom.xml index d2b1b91..9039d8b 100644 --- a/pom.xml +++ b/pom.xml @@ -33,6 +33,8 @@ under the License. 17 UTF-8 5.11.3 + 53.1.0 + 3.25.5 @@ -58,9 +60,21 @@ under the License. 19.0.0 runtime + + com.google.protobuf + protobuf-java + ${protobuf.version} + + + + kr.motd.maven + os-maven-plugin + 1.7.1 + + org.apache.maven.plugins @@ -118,6 +132,7 @@ under the License. NOTICE.txt README.md + CONTRIBUTING.md docs/** .gitignore @@ -138,6 +153,49 @@ under the License. + + com.googlecode.maven-download-plugin + download-maven-plugin + 1.9.0 + + + fetch-datafusion-common-proto + generate-sources + wget + + https://raw.githubusercontent.com/apache/datafusion/${datafusion.version}/datafusion/proto-common/proto/datafusion_common.proto + ${project.build.directory}/proto/datafusion/proto-common/proto + datafusion_common.proto + d6f3368372ea277cc23e26f196994b81616d38599357bb374cbd7eb1760e649a789e4c133d86a395ac701049a500348da2ec039d3f978ac5d8112c2876dded1f + + + + fetch-datafusion-proto + generate-sources + wget + + https://raw.githubusercontent.com/apache/datafusion/${datafusion.version}/datafusion/proto/proto/datafusion.proto + ${project.build.directory}/proto/datafusion/proto/proto + datafusion.proto + c3d162b8e2a418e03f74caceaccfd934af89bb95a12ede13d4cc1701d24c734d74b1e96372142b173db05938dab7f965ad60d476363308c441677a63ea5fbcf7 + + + + + + org.xolstice.maven.plugins + protobuf-maven-plugin + 0.6.1 + + com.google.protobuf:protoc:${protobuf.version}:exe:${os.detected.classifier} + ${project.build.directory}/proto + + + + compile + + + diff --git a/src/test/java/org/apache/datafusion/proto/ProtoGenerationTest.java b/src/test/java/org/apache/datafusion/proto/ProtoGenerationTest.java new file mode 100644 index 0000000..9d99763 --- /dev/null +++ b/src/test/java/org/apache/datafusion/proto/ProtoGenerationTest.java @@ -0,0 +1,41 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.datafusion.proto; + +import static org.junit.jupiter.api.Assertions.assertNotNull; + +import org.apache.datafusion.protobuf.LogicalPlanNode; +import org.junit.jupiter.api.Test; + +/** + * Smoke test: confirms the datafusion-proto schema was downloaded, protoc generated Java sources, + * those sources landed on the compile classpath, and the {@code protobuf-java} runtime resolves. + * + *

Does not exercise any DataFusion plan semantics — those tests arrive with JVM-side plan + * construction. + */ +class ProtoGenerationTest { + + @Test + void generatedClassIsLoadableAndConstructible() { + LogicalPlanNode node = LogicalPlanNode.newBuilder().build(); + assertNotNull(node); + } +} From 058840a4ab7362cdbe38b6d7fa4b03b60bc8955d Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Tue, 12 May 2026 21:35:18 -0600 Subject: [PATCH 09/14] ci: add build workflow that runs make test (#15) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary - Adds `.github/workflows/build.yml` running on pushes to `main` and on PRs targeting `main`. - Sets up Temurin JDK 17 and a stable Rust toolchain, caches Maven (`~/.m2`) and cargo (`native/target`) artifacts, then runs `make test`. ## Rationale The project currently has no CI: `format.yml` was removed in #7 to clear a deadlock when GitHub Actions wasn't yet enabled, and #11 nulled out the stale branch protection rules left behind. With Actions now in place, we need at least one workflow that exercises the build on every PR so regressions surface before merge. `make test` is the canonical full build entry point documented in `README.md` and `CONTRIBUTING.md` — it depends on the `native` target (so it builds the Rust crate first) and then runs the JVM JUnit suite, which is exactly what CI needs to cover. ## What's in this PR - `.github/workflows/build.yml`: single `build` job on `ubuntu-latest`, JDK 17 (Temurin) with Maven cache via `actions/setup-java`, Rust stable via `dtolnay/rust-toolchain`, cargo cache via `Swatinem/rust-cache` scoped to the `native` workspace, then `make test`. ## Not in this PR - Spotless / Apache RAT / clippy / `cargo fmt` checks. These run at Maven's `verify` phase or via separate cargo commands and aren't wired into the Makefile yet; they can be added in a follow-up once we decide whether to extend the Makefile or invoke them directly from CI. - Matrix builds across OS or JDK versions — start minimal, expand if needed. --- .asf.yaml | 2 +- .github/workflows/build.yml | 52 +++++++++++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+), 1 deletion(-) create mode 100644 .github/workflows/build.yml diff --git a/.asf.yaml b/.asf.yaml index cf1630f..175f349 100644 --- a/.asf.yaml +++ b/.asf.yaml @@ -48,7 +48,7 @@ github: issues: true discussions: true protected_branches: - main: ~ + main: {} pull_requests: allow_update_branch: true allow_auto_merge: true diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml new file mode 100644 index 0000000..2db4afc --- /dev/null +++ b/.github/workflows/build.yml @@ -0,0 +1,52 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +name: Build + +on: + push: + branches: [main] + pull_request: + branches: [main] + +permissions: + contents: read + +jobs: + build: + name: make test + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Set up JDK 17 + uses: actions/setup-java@v4 + with: + distribution: temurin + java-version: '17' + cache: maven + + - name: Set up Rust toolchain + uses: dtolnay/rust-toolchain@stable + + - name: Cache cargo build + uses: Swatinem/rust-cache@v2 + with: + workspaces: native + + - name: Build native and run JVM tests + run: make test From 1da32338c44eef5c5325290cc1a52eadf7cca0b3 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Tue, 12 May 2026 21:43:54 -0600 Subject: [PATCH 10/14] ci: replace disallowed third-party actions in build workflow (#16) --- .github/workflows/build.yml | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 2db4afc..de739cc 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -41,12 +41,19 @@ jobs: cache: maven - name: Set up Rust toolchain - uses: dtolnay/rust-toolchain@stable + run: | + rustup update stable + rustup default stable - - name: Cache cargo build - uses: Swatinem/rust-cache@v2 + - name: Cache cargo + uses: actions/cache@v4 with: - workspaces: native + path: | + ~/.cargo/registry + ~/.cargo/git + native/target + key: ${{ runner.os }}-cargo-${{ hashFiles('native/Cargo.lock') }} + restore-keys: ${{ runner.os }}-cargo- - name: Build native and run JVM tests run: make test From 233243cee6f8d5319135a99e9ac70948482b81ca Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Tue, 12 May 2026 23:51:34 -0600 Subject: [PATCH 11/14] feat(parquet): expose ParquetReadOptions via registerParquet and readParquet (#18) --- native/src/lib.rs | 117 +++++++++++++-- .../apache/datafusion/ParquetReadOptions.java | 84 +++++++++++ .../org/apache/datafusion/SessionContext.java | 95 ++++++++++++- .../datafusion/ParquetReadOptionsTest.java | 73 ++++++++++ .../SessionContextParquetOptionsTest.java | 134 ++++++++++++++++++ 5 files changed, 491 insertions(+), 12 deletions(-) create mode 100644 src/main/java/org/apache/datafusion/ParquetReadOptions.java create mode 100644 src/test/java/org/apache/datafusion/ParquetReadOptionsTest.java create mode 100644 src/test/java/org/apache/datafusion/SessionContextParquetOptionsTest.java diff --git a/native/src/lib.rs b/native/src/lib.rs index 947d47f..cc58b7d 100644 --- a/native/src/lib.rs +++ b/native/src/lib.rs @@ -19,14 +19,15 @@ mod errors; use std::sync::{Arc, OnceLock}; -use datafusion::arrow::datatypes::SchemaRef; +use datafusion::arrow::datatypes::{Schema, SchemaRef}; use datafusion::arrow::ffi_stream::FFI_ArrowArrayStream; +use datafusion::arrow::ipc::reader::StreamReader; use datafusion::arrow::record_batch::RecordBatchIterator; use datafusion::dataframe::DataFrame; use datafusion::error::DataFusionError; use datafusion::prelude::{ParquetReadOptions, SessionContext}; -use jni::objects::{JClass, JString}; -use jni::sys::jlong; +use jni::objects::{JByteArray, JClass, JString}; +use jni::sys::{jboolean, jlong}; use jni::JNIEnv; use tokio::runtime::Runtime; @@ -129,13 +130,61 @@ pub extern "system" fn Java_org_apache_datafusion_SessionContext_closeSessionCon }) } +#[allow(clippy::too_many_arguments)] +fn with_parquet_options( + env: &mut JNIEnv, + file_extension: JString, + parquet_pruning_set: jboolean, + parquet_pruning_value: jboolean, + skip_metadata_set: jboolean, + skip_metadata_value: jboolean, + metadata_size_hint: jlong, + schema_ipc_bytes: JByteArray, + f: impl FnOnce(ParquetReadOptions) -> JniResult, +) -> JniResult { + let file_ext: String = env.get_string(&file_extension)?.into(); + + let schema: Option = if !schema_ipc_bytes.is_null() { + let bytes: Vec = env.convert_byte_array(&schema_ipc_bytes)?; + let reader = StreamReader::try_new(std::io::Cursor::new(bytes), None)?; + Some((*reader.schema()).clone()) + } else { + None + }; + + let mut opts = ParquetReadOptions::default().file_extension(&file_ext); + if parquet_pruning_set != 0 { + opts = opts.parquet_pruning(parquet_pruning_value != 0); + } + if skip_metadata_set != 0 { + opts = opts.skip_metadata(skip_metadata_value != 0); + } + if metadata_size_hint >= 0 { + opts = opts.metadata_size_hint(Some(metadata_size_hint as usize)); + } + if let Some(ref s) = schema { + opts = opts.schema(s); + } + + f(opts) +} + #[no_mangle] -pub extern "system" fn Java_org_apache_datafusion_SessionContext_registerParquet<'local>( +pub extern "system" fn Java_org_apache_datafusion_SessionContext_registerParquetWithOptions< + 'local, +>( mut env: JNIEnv<'local>, _class: JClass<'local>, handle: jlong, name: JString<'local>, path: JString<'local>, + file_extension: JString<'local>, + parquet_pruning_set: jboolean, + parquet_pruning_value: jboolean, + skip_metadata_set: jboolean, + skip_metadata_value: jboolean, + metadata_size_hint: jlong, + schema_ipc_bytes: JByteArray<'local>, ) { try_unwrap_or_throw(&mut env, (), |env| -> JniResult<()> { if handle == 0 { @@ -144,11 +193,59 @@ pub extern "system" fn Java_org_apache_datafusion_SessionContext_registerParquet let ctx = unsafe { &*(handle as *const SessionContext) }; let name: String = env.get_string(&name)?.into(); let path: String = env.get_string(&path)?.into(); - runtime().block_on(async { - ctx.register_parquet(&name, &path, ParquetReadOptions::default()) - .await?; - Ok::<(), DataFusionError>(()) - })?; - Ok(()) + with_parquet_options( + env, + file_extension, + parquet_pruning_set, + parquet_pruning_value, + skip_metadata_set, + skip_metadata_value, + metadata_size_hint, + schema_ipc_bytes, + |opts| { + runtime().block_on(async { + ctx.register_parquet(&name, &path, opts).await?; + Ok::<(), DataFusionError>(()) + })?; + Ok(()) + }, + ) + }) +} + +#[no_mangle] +pub extern "system" fn Java_org_apache_datafusion_SessionContext_readParquetWithOptions<'local>( + mut env: JNIEnv<'local>, + _class: JClass<'local>, + handle: jlong, + path: JString<'local>, + file_extension: JString<'local>, + parquet_pruning_set: jboolean, + parquet_pruning_value: jboolean, + skip_metadata_set: jboolean, + skip_metadata_value: jboolean, + metadata_size_hint: jlong, + schema_ipc_bytes: JByteArray<'local>, +) -> jlong { + try_unwrap_or_throw(&mut env, 0, |env| -> JniResult { + if handle == 0 { + return Err("SessionContext handle is null".into()); + } + let ctx = unsafe { &*(handle as *const SessionContext) }; + let path: String = env.get_string(&path)?.into(); + with_parquet_options( + env, + file_extension, + parquet_pruning_set, + parquet_pruning_value, + skip_metadata_set, + skip_metadata_value, + metadata_size_hint, + schema_ipc_bytes, + |opts| { + let df = runtime().block_on(ctx.read_parquet(path, opts))?; + Ok(Box::into_raw(Box::new(df)) as jlong) + }, + ) }) } diff --git a/src/main/java/org/apache/datafusion/ParquetReadOptions.java b/src/main/java/org/apache/datafusion/ParquetReadOptions.java new file mode 100644 index 0000000..b29d762 --- /dev/null +++ b/src/main/java/org/apache/datafusion/ParquetReadOptions.java @@ -0,0 +1,84 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.datafusion; + +import org.apache.arrow.vector.types.pojo.Schema; + +/** + * Configuration knobs for parquet sources passed to {@link SessionContext#registerParquet(String, + * String, ParquetReadOptions)} and {@link SessionContext#readParquet(String, ParquetReadOptions)}. + * + *

Mirrors a subset of DataFusion's {@code ParquetReadOptions}. All setters return {@code this} + * for fluent chaining. Defaults: {@code fileExtension = ".parquet"}; all other fields {@code null} + * (meaning the SessionConfig default is used, or the schema is inferred from the file). + */ +public final class ParquetReadOptions { + + private String fileExtension = ".parquet"; + private Boolean parquetPruning; + private Boolean skipMetadata; + private Long metadataSizeHint; + private Schema schema; + + public ParquetReadOptions fileExtension(String ext) { + this.fileExtension = ext; + return this; + } + + public ParquetReadOptions parquetPruning(boolean v) { + this.parquetPruning = v; + return this; + } + + public ParquetReadOptions skipMetadata(boolean v) { + this.skipMetadata = v; + return this; + } + + public ParquetReadOptions metadataSizeHint(long bytes) { + this.metadataSizeHint = bytes; + return this; + } + + public ParquetReadOptions schema(Schema schema) { + this.schema = schema; + return this; + } + + String fileExtension() { + return fileExtension; + } + + Boolean parquetPruning() { + return parquetPruning; + } + + Boolean skipMetadata() { + return skipMetadata; + } + + Long metadataSizeHint() { + return metadataSizeHint; + } + + Schema schema() { + return schema; + } +} diff --git a/src/main/java/org/apache/datafusion/SessionContext.java b/src/main/java/org/apache/datafusion/SessionContext.java index ac1a45e..fb79d43 100644 --- a/src/main/java/org/apache/datafusion/SessionContext.java +++ b/src/main/java/org/apache/datafusion/SessionContext.java @@ -19,6 +19,16 @@ package org.apache.datafusion; +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.nio.channels.Channels; + +import org.apache.arrow.memory.BufferAllocator; +import org.apache.arrow.memory.RootAllocator; +import org.apache.arrow.vector.VectorSchemaRoot; +import org.apache.arrow.vector.ipc.ArrowStreamWriter; +import org.apache.arrow.vector.types.pojo.Schema; + /** * A DataFusion session context. * @@ -54,10 +64,70 @@ public DataFrame sql(String query) { } public void registerParquet(String name, String path) { + registerParquet(name, path, new ParquetReadOptions()); + } + + /** + * Register a parquet file as a table with the supplied {@link ParquetReadOptions}. + * + * @throws RuntimeException if registration fails (path not found, schema mismatch, etc.). + */ + public void registerParquet(String name, String path, ParquetReadOptions options) { + if (nativeHandle == 0) { + throw new IllegalStateException("SessionContext is closed"); + } + registerParquetWithOptions( + nativeHandle, + name, + path, + options.fileExtension(), + options.parquetPruning() != null, + options.parquetPruning() != null && options.parquetPruning(), + options.skipMetadata() != null, + options.skipMetadata() != null && options.skipMetadata(), + options.metadataSizeHint() != null ? options.metadataSizeHint() : -1L, + options.schema() != null ? serializeSchemaIpc(options.schema()) : null); + } + + /** Read a parquet file as a {@link DataFrame} without registering it. */ + public DataFrame readParquet(String path) { + return readParquet(path, new ParquetReadOptions()); + } + + /** + * Read a parquet file as a {@link DataFrame} with the supplied {@link ParquetReadOptions}. + * + * @throws RuntimeException if the read fails. + */ + public DataFrame readParquet(String path, ParquetReadOptions options) { if (nativeHandle == 0) { throw new IllegalStateException("SessionContext is closed"); } - registerParquet(nativeHandle, name, path); + long dfHandle = + readParquetWithOptions( + nativeHandle, + path, + options.fileExtension(), + options.parquetPruning() != null, + options.parquetPruning() != null && options.parquetPruning(), + options.skipMetadata() != null, + options.skipMetadata() != null && options.skipMetadata(), + options.metadataSizeHint() != null ? options.metadataSizeHint() : -1L, + options.schema() != null ? serializeSchemaIpc(options.schema()) : null); + return new DataFrame(dfHandle); + } + + private static byte[] serializeSchemaIpc(Schema schema) { + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + try (BufferAllocator allocator = new RootAllocator(); + VectorSchemaRoot root = VectorSchemaRoot.create(schema, allocator); + ArrowStreamWriter writer = new ArrowStreamWriter(root, null, Channels.newChannel(baos))) { + writer.start(); + writer.end(); + } catch (IOException e) { + throw new RuntimeException("Failed to serialize Arrow schema for JNI", e); + } + return baos.toByteArray(); } @Override @@ -72,7 +142,28 @@ public void close() { private static native long createDataFrame(long handle, String sql); - private static native void registerParquet(long handle, String name, String path); + private static native void registerParquetWithOptions( + long handle, + String name, + String path, + String fileExtension, + boolean parquetPruningSet, + boolean parquetPruningValue, + boolean skipMetadataSet, + boolean skipMetadataValue, + long metadataSizeHint, + byte[] schemaIpcBytes); + + private static native long readParquetWithOptions( + long handle, + String path, + String fileExtension, + boolean parquetPruningSet, + boolean parquetPruningValue, + boolean skipMetadataSet, + boolean skipMetadataValue, + long metadataSizeHint, + byte[] schemaIpcBytes); private static native void closeSessionContext(long handle); } diff --git a/src/test/java/org/apache/datafusion/ParquetReadOptionsTest.java b/src/test/java/org/apache/datafusion/ParquetReadOptionsTest.java new file mode 100644 index 0000000..3eca311 --- /dev/null +++ b/src/test/java/org/apache/datafusion/ParquetReadOptionsTest.java @@ -0,0 +1,73 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.datafusion; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertSame; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.util.List; + +import org.apache.arrow.vector.types.pojo.ArrowType; +import org.apache.arrow.vector.types.pojo.Field; +import org.apache.arrow.vector.types.pojo.FieldType; +import org.apache.arrow.vector.types.pojo.Schema; +import org.junit.jupiter.api.Test; + +class ParquetReadOptionsTest { + + @Test + void defaultsMatchDataFusion() { + ParquetReadOptions opts = new ParquetReadOptions(); + assertEquals(".parquet", opts.fileExtension()); + assertNull(opts.parquetPruning()); + assertNull(opts.skipMetadata()); + assertNull(opts.metadataSizeHint()); + assertNull(opts.schema()); + } + + @Test + void fluentSettersChainAndMutate() { + Schema schema = + new Schema(List.of(new Field("x", FieldType.nullable(new ArrowType.Int(32, true)), null))); + + ParquetReadOptions opts = + new ParquetReadOptions() + .fileExtension(".parq") + .parquetPruning(true) + .skipMetadata(false) + .metadataSizeHint(1_048_576L) + .schema(schema); + + assertEquals(".parq", opts.fileExtension()); + assertEquals(Boolean.TRUE, opts.parquetPruning()); + assertEquals(Boolean.FALSE, opts.skipMetadata()); + assertEquals(Long.valueOf(1_048_576L), opts.metadataSizeHint()); + assertTrue(opts.schema() == schema); + } + + @Test + void schemaSetterRetainsReferenceIdentity() { + Schema schema = new Schema(List.of()); + ParquetReadOptions opts = new ParquetReadOptions().schema(schema); + assertSame(schema, opts.schema()); + } +} diff --git a/src/test/java/org/apache/datafusion/SessionContextParquetOptionsTest.java b/src/test/java/org/apache/datafusion/SessionContextParquetOptionsTest.java new file mode 100644 index 0000000..9b79327 --- /dev/null +++ b/src/test/java/org/apache/datafusion/SessionContextParquetOptionsTest.java @@ -0,0 +1,134 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.datafusion; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.StandardCopyOption; +import java.util.List; + +import org.apache.arrow.memory.BufferAllocator; +import org.apache.arrow.memory.RootAllocator; +import org.apache.arrow.vector.BigIntVector; +import org.apache.arrow.vector.VectorSchemaRoot; +import org.apache.arrow.vector.ipc.ArrowReader; +import org.apache.arrow.vector.types.pojo.ArrowType; +import org.apache.arrow.vector.types.pojo.Field; +import org.apache.arrow.vector.types.pojo.FieldType; +import org.apache.arrow.vector.types.pojo.Schema; +import org.junit.jupiter.api.Assumptions; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +class SessionContextParquetOptionsTest { + + @Test + void readParquetWithDefaultOptionsCountsAllRows() throws Exception { + Path lineitem = Path.of("tpch-data/sf1/lineitem.parquet"); + Assumptions.assumeTrue( + Files.exists(lineitem), "TPC-H SF1 data not found; run `make tpch-data` first"); + + try (BufferAllocator allocator = new RootAllocator(); + SessionContext ctx = new SessionContext(); + DataFrame df = ctx.readParquet(lineitem.toAbsolutePath().toString()); + ArrowReader reader = df.collect(allocator)) { + long total = 0; + while (reader.loadNextBatch()) { + VectorSchemaRoot root = reader.getVectorSchemaRoot(); + total += root.getRowCount(); + } + assertEquals(6_001_215L, total); + assertTrue(total > 0); + } + } + + @Test + void registerParquetWithOptionsRespectsCustomExtension(@TempDir Path tempDir) throws Exception { + Path lineitem = Path.of("tpch-data/sf1/lineitem.parquet"); + Assumptions.assumeTrue( + Files.exists(lineitem), "TPC-H SF1 data not found; run `make tpch-data` first"); + Path renamed = tempDir.resolve("lineitem.parq"); + Files.copy(lineitem, renamed, StandardCopyOption.REPLACE_EXISTING); + + try (BufferAllocator allocator = new RootAllocator(); + SessionContext ctx = new SessionContext()) { + ctx.registerParquet( + "t", + renamed.toAbsolutePath().toString(), + new ParquetReadOptions().fileExtension(".parq")); + try (DataFrame df = ctx.sql("SELECT COUNT(*) FROM t"); + ArrowReader reader = df.collect(allocator)) { + assertTrue(reader.loadNextBatch()); + VectorSchemaRoot root = reader.getVectorSchemaRoot(); + assertEquals(1, root.getRowCount()); + BigIntVector count = (BigIntVector) root.getVector(0); + assertEquals(6_001_215L, count.get(0)); + } + } + } + + @Test + void readParquetWithExplicitSchemaUsesProvidedSchema() throws Exception { + Path lineitem = Path.of("tpch-data/sf1/lineitem.parquet"); + Assumptions.assumeTrue( + Files.exists(lineitem), "TPC-H SF1 data not found; run `make tpch-data` first"); + + Schema custom = + new Schema( + List.of( + new Field("l_orderkey", FieldType.notNullable(new ArrowType.Int(64, true)), null))); + + try (BufferAllocator allocator = new RootAllocator(); + SessionContext ctx = new SessionContext(); + DataFrame df = + ctx.readParquet( + lineitem.toAbsolutePath().toString(), new ParquetReadOptions().schema(custom)); + ArrowReader reader = df.collect(allocator)) { + assertTrue(reader.loadNextBatch()); + Schema observed = reader.getVectorSchemaRoot().getSchema(); + assertEquals(1, observed.getFields().size()); + assertEquals("l_orderkey", observed.getFields().get(0).getName()); + } + } + + @Test + void registerParquetWithMetadataSizeHintIsAccepted() throws Exception { + Path lineitem = Path.of("tpch-data/sf1/lineitem.parquet"); + Assumptions.assumeTrue( + Files.exists(lineitem), "TPC-H SF1 data not found; run `make tpch-data` first"); + + try (BufferAllocator allocator = new RootAllocator(); + SessionContext ctx = new SessionContext()) { + ctx.registerParquet( + "t", + lineitem.toAbsolutePath().toString(), + new ParquetReadOptions().metadataSizeHint(1L << 20)); + try (DataFrame df = ctx.sql("SELECT COUNT(*) FROM t"); + ArrowReader reader = df.collect(allocator)) { + assertTrue(reader.loadNextBatch()); + BigIntVector count = (BigIntVector) reader.getVectorSchemaRoot().getVector(0); + assertEquals(6_001_215L, count.get(0)); + } + } + } +} From 717b735d3ac2204808c47448d600e487ac968b9c Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Wed, 13 May 2026 00:33:41 -0600 Subject: [PATCH 12/14] feat(dataframe): add select/filter/count/show methods (#19) --- native/src/lib.rs | 99 ++++++++- .../java/org/apache/datafusion/DataFrame.java | 56 +++++ .../DataFrameTransformationsTest.java | 196 ++++++++++++++++++ 3 files changed, 349 insertions(+), 2 deletions(-) create mode 100644 src/test/java/org/apache/datafusion/DataFrameTransformationsTest.java diff --git a/native/src/lib.rs b/native/src/lib.rs index cc58b7d..42ca5f5 100644 --- a/native/src/lib.rs +++ b/native/src/lib.rs @@ -26,8 +26,8 @@ use datafusion::arrow::record_batch::RecordBatchIterator; use datafusion::dataframe::DataFrame; use datafusion::error::DataFusionError; use datafusion::prelude::{ParquetReadOptions, SessionContext}; -use jni::objects::{JByteArray, JClass, JString}; -use jni::sys::{jboolean, jlong}; +use jni::objects::{JByteArray, JClass, JObjectArray, JString}; +use jni::sys::{jboolean, jint, jlong}; use jni::JNIEnv; use tokio::runtime::Runtime; @@ -98,6 +98,101 @@ pub extern "system" fn Java_org_apache_datafusion_DataFrame_collectDataFrame<'lo }) } +#[no_mangle] +pub extern "system" fn Java_org_apache_datafusion_DataFrame_countRows<'local>( + mut env: JNIEnv<'local>, + _class: JClass<'local>, + handle: jlong, +) -> jlong { + try_unwrap_or_throw(&mut env, 0, |_env| -> JniResult { + if handle == 0 { + return Err("DataFrame handle is null".into()); + } + let df = unsafe { &*(handle as *const DataFrame) }.clone(); + let n = runtime().block_on(async { df.count().await })?; + Ok(n as jlong) + }) +} + +#[no_mangle] +pub extern "system" fn Java_org_apache_datafusion_DataFrame_showDataFrame<'local>( + mut env: JNIEnv<'local>, + _class: JClass<'local>, + handle: jlong, +) { + try_unwrap_or_throw(&mut env, (), |_env| -> JniResult<()> { + if handle == 0 { + return Err("DataFrame handle is null".into()); + } + let df = unsafe { &*(handle as *const DataFrame) }.clone(); + runtime().block_on(async { df.show().await })?; + Ok(()) + }) +} + +#[no_mangle] +pub extern "system" fn Java_org_apache_datafusion_DataFrame_showDataFrameWithLimit<'local>( + mut env: JNIEnv<'local>, + _class: JClass<'local>, + handle: jlong, + limit: jint, +) { + try_unwrap_or_throw(&mut env, (), |_env| -> JniResult<()> { + if handle == 0 { + return Err("DataFrame handle is null".into()); + } + let df = unsafe { &*(handle as *const DataFrame) }.clone(); + runtime().block_on(async { df.show_limit(limit as usize).await })?; + Ok(()) + }) +} + +#[no_mangle] +pub extern "system" fn Java_org_apache_datafusion_DataFrame_selectColumns<'local>( + mut env: JNIEnv<'local>, + _class: JClass<'local>, + handle: jlong, + column_names: JObjectArray<'local>, +) -> jlong { + try_unwrap_or_throw(&mut env, 0, |env| -> JniResult { + if handle == 0 { + return Err("DataFrame handle is null".into()); + } + let df = unsafe { &*(handle as *const DataFrame) }.clone(); + + let len = env.get_array_length(&column_names)?; + let mut owned: Vec = Vec::with_capacity(len as usize); + for i in 0..len { + let elem = env.get_object_array_element(&column_names, i)?; + let jstr: JString = elem.into(); + owned.push(env.get_string(&jstr)?.into()); + } + let refs: Vec<&str> = owned.iter().map(String::as_str).collect(); + + let new_df = df.select_columns(&refs)?; + Ok(Box::into_raw(Box::new(new_df)) as jlong) + }) +} + +#[no_mangle] +pub extern "system" fn Java_org_apache_datafusion_DataFrame_filterRows<'local>( + mut env: JNIEnv<'local>, + _class: JClass<'local>, + handle: jlong, + predicate: JString<'local>, +) -> jlong { + try_unwrap_or_throw(&mut env, 0, |env| -> JniResult { + if handle == 0 { + return Err("DataFrame handle is null".into()); + } + let df = unsafe { &*(handle as *const DataFrame) }.clone(); + let predicate: String = env.get_string(&predicate)?.into(); + let expr = df.parse_sql_expr(&predicate)?; + let new_df = df.filter(expr)?; + Ok(Box::into_raw(Box::new(new_df)) as jlong) + }) +} + #[no_mangle] pub extern "system" fn Java_org_apache_datafusion_DataFrame_closeDataFrame<'local>( mut env: JNIEnv<'local>, diff --git a/src/main/java/org/apache/datafusion/DataFrame.java b/src/main/java/org/apache/datafusion/DataFrame.java index f285879..0bd77e7 100644 --- a/src/main/java/org/apache/datafusion/DataFrame.java +++ b/src/main/java/org/apache/datafusion/DataFrame.java @@ -70,6 +70,52 @@ public ArrowReader collect(BufferAllocator allocator) { } } + /** Execute the plan and return the number of rows. */ + public long count() { + if (nativeHandle == 0) { + throw new IllegalStateException("DataFrame is closed or already collected"); + } + return countRows(nativeHandle); + } + + /** Execute the plan and print formatted batches to native stdout. */ + public void show() { + if (nativeHandle == 0) { + throw new IllegalStateException("DataFrame is closed or already collected"); + } + showDataFrame(nativeHandle); + } + + /** Execute the plan and print the first {@code limit} rows to native stdout. */ + public void show(int limit) { + if (nativeHandle == 0) { + throw new IllegalStateException("DataFrame is closed or already collected"); + } + showDataFrameWithLimit(nativeHandle, limit); + } + + /** + * Project the listed columns into a new DataFrame. The receiver remains usable and must still be + * closed independently. + */ + public DataFrame select(String... columnNames) { + if (nativeHandle == 0) { + throw new IllegalStateException("DataFrame is closed or already collected"); + } + return new DataFrame(selectColumns(nativeHandle, columnNames)); + } + + /** + * Apply a SQL predicate to produce a filtered DataFrame. The predicate is parsed against this + * DataFrame's own schema. The receiver remains usable and must still be closed independently. + */ + public DataFrame filter(String predicate) { + if (nativeHandle == 0) { + throw new IllegalStateException("DataFrame is closed or already collected"); + } + return new DataFrame(filterRows(nativeHandle, predicate)); + } + @Override public void close() { if (nativeHandle != 0) { @@ -81,4 +127,14 @@ public void close() { private static native void collectDataFrame(long handle, long ffiStreamAddr); private static native void closeDataFrame(long handle); + + private static native long countRows(long handle); + + private static native void showDataFrame(long handle); + + private static native void showDataFrameWithLimit(long handle, int limit); + + private static native long selectColumns(long handle, String[] columnNames); + + private static native long filterRows(long handle, String predicate); } diff --git a/src/test/java/org/apache/datafusion/DataFrameTransformationsTest.java b/src/test/java/org/apache/datafusion/DataFrameTransformationsTest.java new file mode 100644 index 0000000..09c7912 --- /dev/null +++ b/src/test/java/org/apache/datafusion/DataFrameTransformationsTest.java @@ -0,0 +1,196 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.datafusion; + +import static org.junit.jupiter.api.Assertions.assertArrayEquals; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.nio.file.Files; +import java.nio.file.Path; + +import org.apache.arrow.memory.BufferAllocator; +import org.apache.arrow.memory.RootAllocator; +import org.apache.arrow.vector.BigIntVector; +import org.apache.arrow.vector.VectorSchemaRoot; +import org.apache.arrow.vector.ipc.ArrowReader; +import org.junit.jupiter.api.Assumptions; +import org.junit.jupiter.api.Test; + +class DataFrameTransformationsTest { + @Test + void countReturnsRowCount() { + try (SessionContext ctx = new SessionContext(); + DataFrame df = ctx.sql("SELECT * FROM (VALUES (1), (2), (3)) AS t(x)")) { + assertEquals(3L, df.count()); + } + } + + @Test + void showDoesNotThrow() { + try (SessionContext ctx = new SessionContext(); + DataFrame df = ctx.sql("SELECT * FROM (VALUES (1), (2)) AS t(x)")) { + df.show(); + } + } + + @Test + void showWithLimitDoesNotThrow() { + try (SessionContext ctx = new SessionContext(); + DataFrame df = ctx.sql("SELECT * FROM (VALUES (1), (2), (3)) AS t(x)")) { + df.show(0); + df.show(1); + df.show(1_000_000); + } + } + + @Test + void selectProjectsAndReordersColumns() throws Exception { + try (BufferAllocator allocator = new RootAllocator(); + SessionContext ctx = new SessionContext(); + DataFrame source = ctx.sql("SELECT 1 AS a, 2 AS b, 3 AS c"); + DataFrame projected = source.select("b", "a"); + ArrowReader reader = projected.collect(allocator)) { + assertTrue(reader.loadNextBatch()); + VectorSchemaRoot root = reader.getVectorSchemaRoot(); + assertEquals(1, root.getRowCount()); + assertArrayEquals( + new String[] {"b", "a"}, + root.getSchema().getFields().stream().map(f -> f.getName()).toArray(String[]::new)); + } + } + + @Test + void selectIsNonDestructive() { + try (SessionContext ctx = new SessionContext(); + DataFrame source = ctx.sql("SELECT 1 AS a, 2 AS b")) { + try (DataFrame first = source.select("a")) { + assertEquals(1L, first.count()); + } + try (DataFrame second = source.select("b")) { + assertEquals(1L, second.count()); + } + assertEquals(1L, source.count()); + } + } + + @Test + void filterRemovesRows() { + try (SessionContext ctx = new SessionContext(); + DataFrame source = ctx.sql("SELECT * FROM (VALUES (1), (2), (3), (4)) AS t(x)"); + DataFrame filtered = source.filter("x > 2")) { + assertEquals(2L, filtered.count()); + } + } + + @Test + void filterIsNonDestructive() { + try (SessionContext ctx = new SessionContext(); + DataFrame source = ctx.sql("SELECT * FROM (VALUES (1), (2), (3), (4)) AS t(x)")) { + try (DataFrame filtered = source.filter("x > 2")) { + assertEquals(2L, filtered.count()); + } + assertEquals(4L, source.count()); + } + } + + @Test + void chainFilterSelectCount() { + try (SessionContext ctx = new SessionContext(); + DataFrame source = ctx.sql("SELECT 1 AS a, 2 AS b UNION ALL SELECT 10 AS a, 20 AS b"); + DataFrame chained = source.filter("a > 5").select("b")) { + assertEquals(1L, chained.count()); + } + } + + @Test + void methodsThrowAfterClose() { + try (SessionContext ctx = new SessionContext()) { + DataFrame df = ctx.sql("SELECT 1 AS x"); + df.close(); + assertThrows(IllegalStateException.class, () -> df.select("x")); + assertThrows(IllegalStateException.class, () -> df.filter("x > 0")); + assertThrows(IllegalStateException.class, df::count); + assertThrows(IllegalStateException.class, df::show); + assertThrows(IllegalStateException.class, () -> df.show(5)); + } + } + + @Test + void methodsThrowAfterCollect() throws Exception { + try (BufferAllocator allocator = new RootAllocator(); + SessionContext ctx = new SessionContext(); + DataFrame df = ctx.sql("SELECT 1 AS x")) { + try (ArrowReader reader = df.collect(allocator)) { + assertTrue(reader.loadNextBatch()); + } + assertThrows(IllegalStateException.class, () -> df.select("x")); + assertThrows(IllegalStateException.class, () -> df.filter("x > 0")); + assertThrows(IllegalStateException.class, df::count); + assertThrows(IllegalStateException.class, df::show); + assertThrows(IllegalStateException.class, () -> df.show(5)); + } + } + + @Test + void selectInvalidColumnThrows() { + try (SessionContext ctx = new SessionContext(); + DataFrame df = ctx.sql("SELECT 1 AS x")) { + assertThrows(RuntimeException.class, () -> df.select("not_a_column")); + } + } + + @Test + void filterMalformedPredicateThrows() { + try (SessionContext ctx = new SessionContext(); + DataFrame df = ctx.sql("SELECT 1 AS x")) { + assertThrows(RuntimeException.class, () -> df.filter("this is not sql")); + } + } + + @Test + void lineitemFilterCountAgainstSqlBaseline() throws Exception { + Path lineitem = Path.of("tpch-data/sf1/lineitem.parquet"); + Assumptions.assumeTrue( + Files.exists(lineitem), "TPC-H SF1 data not found; run `make tpch-data` first"); + + try (SessionContext ctx = new SessionContext()) { + ctx.registerParquet("lineitem", lineitem.toAbsolutePath().toString()); + + long viaDataFrame; + try (DataFrame df = ctx.sql("SELECT * FROM lineitem"); + DataFrame filtered = df.filter("l_orderkey < 100")) { + viaDataFrame = filtered.count(); + } + + long viaSql; + try (BufferAllocator allocator = new RootAllocator(); + DataFrame df = ctx.sql("SELECT COUNT(*) FROM lineitem WHERE l_orderkey < 100"); + ArrowReader reader = df.collect(allocator)) { + assertTrue(reader.loadNextBatch()); + VectorSchemaRoot root = reader.getVectorSchemaRoot(); + viaSql = ((BigIntVector) root.getVector(0)).get(0); + } + + assertEquals(viaSql, viaDataFrame); + } + } +} From 757a078c3ed70c0627b29d6ccf081eaf281761bc Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Wed, 13 May 2026 00:33:54 -0600 Subject: [PATCH 13/14] feat(proto): execute Java-built protobuf plans via SessionContext (#13) --- native/Cargo.lock | 64 ++++++ native/Cargo.toml | 2 + native/src/lib.rs | 3 +- native/src/proto.rs | 81 +++++++ .../org/apache/datafusion/SessionContext.java | 44 ++++ .../datafusion/proto/SchemaConverter.java | 212 ++++++++++++++++++ .../datafusion/proto/SchemaConverterTest.java | 107 +++++++++ .../proto/SessionContextProtoTest.java | 188 ++++++++++++++++ 8 files changed, 700 insertions(+), 1 deletion(-) create mode 100644 native/src/proto.rs create mode 100644 src/main/java/org/apache/datafusion/proto/SchemaConverter.java create mode 100644 src/test/java/org/apache/datafusion/proto/SchemaConverterTest.java create mode 100644 src/test/java/org/apache/datafusion/proto/SessionContextProtoTest.java diff --git a/native/Cargo.lock b/native/Cargo.lock index 5ebef8e..004fa98 100644 --- a/native/Cargo.lock +++ b/native/Cargo.lock @@ -1140,7 +1140,9 @@ version = "0.1.0" dependencies = [ "arrow", "datafusion", + "datafusion-proto", "jni", + "prost", "tokio", ] @@ -1282,6 +1284,45 @@ dependencies = [ "tokio", ] +[[package]] +name = "datafusion-proto" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6a387aaef949dc16bb6abc81bd1af850ec7449183aef011214f9724957495738" +dependencies = [ + "arrow", + "chrono", + "datafusion-catalog", + "datafusion-catalog-listing", + "datafusion-common", + "datafusion-datasource", + "datafusion-datasource-arrow", + "datafusion-datasource-csv", + "datafusion-datasource-json", + "datafusion-datasource-parquet", + "datafusion-execution", + "datafusion-expr", + "datafusion-functions-table", + "datafusion-physical-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-proto-common", + "object_store", + "prost", + "rand", +] + +[[package]] +name = "datafusion-proto-common" +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "16e614c7c53a9c304c6a850b821010bb492e57300311835f1180613f9d2c63d9" +dependencies = [ + "arrow", + "datafusion-common", + "prost", +] + [[package]] name = "datafusion-pruning" version = "53.1.0" @@ -2281,6 +2322,29 @@ dependencies = [ "unicode-ident", ] +[[package]] +name = "prost" +version = "0.14.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d2ea70524a2f82d518bce41317d0fae74151505651af45faf1ffbd6fd33f0568" +dependencies = [ + "bytes", + "prost-derive", +] + +[[package]] +name = "prost-derive" +version = "0.14.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "27c6023962132f4b30eb4c172c91ce92d933da334c59c23cddee82358ddafb0b" +dependencies = [ + "anyhow", + "itertools", + "proc-macro2", + "quote", + "syn", +] + [[package]] name = "psm" version = "0.1.31" diff --git a/native/Cargo.toml b/native/Cargo.toml index 033fb4c..f2a6818 100644 --- a/native/Cargo.toml +++ b/native/Cargo.toml @@ -27,5 +27,7 @@ crate-type = ["cdylib"] [dependencies] arrow = { version = "58", features = ["ffi"] } datafusion = "53.1.0" +datafusion-proto = "53.1.0" jni = "0.21" +prost = "0.14" tokio = { version = "1", features = ["rt-multi-thread"] } diff --git a/native/src/lib.rs b/native/src/lib.rs index 42ca5f5..463d075 100644 --- a/native/src/lib.rs +++ b/native/src/lib.rs @@ -16,6 +16,7 @@ // under the License. mod errors; +mod proto; use std::sync::{Arc, OnceLock}; @@ -33,7 +34,7 @@ use tokio::runtime::Runtime; use crate::errors::{try_unwrap_or_throw, JniResult}; -fn runtime() -> &'static Runtime { +pub(crate) fn runtime() -> &'static Runtime { static RT: OnceLock = OnceLock::new(); RT.get_or_init(|| Runtime::new().expect("failed to create Tokio runtime")) } diff --git a/native/src/proto.rs b/native/src/proto.rs new file mode 100644 index 0000000..429a91e --- /dev/null +++ b/native/src/proto.rs @@ -0,0 +1,81 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use std::sync::Arc; + +use datafusion::arrow::datatypes::SchemaRef; +use datafusion::arrow::ipc::writer::StreamWriter; +use datafusion::dataframe::DataFrame; +use datafusion::prelude::SessionContext; +use datafusion_proto::logical_plan::{AsLogicalPlan, DefaultLogicalExtensionCodec}; +use datafusion_proto::protobuf::LogicalPlanNode; +use jni::objects::{JByteArray, JClass, JString}; +use jni::sys::{jbyteArray, jlong}; +use jni::JNIEnv; +use prost::Message; + +use crate::errors::{try_unwrap_or_throw, JniResult}; +use crate::runtime; + +#[no_mangle] +pub extern "system" fn Java_org_apache_datafusion_SessionContext_createDataFrameFromProto<'local>( + mut env: JNIEnv<'local>, + _class: JClass<'local>, + handle: jlong, + plan_bytes: JByteArray<'local>, +) -> jlong { + try_unwrap_or_throw(&mut env, 0, |env| -> JniResult { + if handle == 0 { + return Err("SessionContext handle is null".into()); + } + let ctx = unsafe { &*(handle as *const SessionContext) }; + let bytes: Vec = env.convert_byte_array(&plan_bytes)?; + let node = LogicalPlanNode::decode(bytes.as_slice())?; + let codec = DefaultLogicalExtensionCodec {}; + let task_ctx = ctx.task_ctx(); + let plan = node.try_into_logical_plan(task_ctx.as_ref(), &codec)?; + let df: DataFrame = runtime().block_on(ctx.execute_logical_plan(plan))?; + Ok(Box::into_raw(Box::new(df)) as jlong) + }) +} + +#[no_mangle] +pub extern "system" fn Java_org_apache_datafusion_SessionContext_tableSchemaIpc<'local>( + mut env: JNIEnv<'local>, + _class: JClass<'local>, + handle: jlong, + name: JString<'local>, +) -> jbyteArray { + try_unwrap_or_throw(&mut env, std::ptr::null_mut(), |env| -> JniResult { + if handle == 0 { + return Err("SessionContext handle is null".into()); + } + let ctx = unsafe { &*(handle as *const SessionContext) }; + let name: String = env.get_string(&name)?.into(); + + let df = runtime().block_on(ctx.table(name.as_str()))?; + let schema: SchemaRef = Arc::new(df.schema().as_arrow().clone()); + + let mut buf: Vec = Vec::new(); + { + let mut writer = StreamWriter::try_new(&mut buf, schema.as_ref())?; + writer.finish()?; + } + let arr = env.byte_array_from_slice(&buf)?; + Ok(arr.into_raw()) + }) +} diff --git a/src/main/java/org/apache/datafusion/SessionContext.java b/src/main/java/org/apache/datafusion/SessionContext.java index fb79d43..823ee13 100644 --- a/src/main/java/org/apache/datafusion/SessionContext.java +++ b/src/main/java/org/apache/datafusion/SessionContext.java @@ -19,6 +19,7 @@ package org.apache.datafusion; +import java.io.ByteArrayInputStream; import java.io.ByteArrayOutputStream; import java.io.IOException; import java.nio.channels.Channels; @@ -27,6 +28,8 @@ import org.apache.arrow.memory.RootAllocator; import org.apache.arrow.vector.VectorSchemaRoot; import org.apache.arrow.vector.ipc.ArrowStreamWriter; +import org.apache.arrow.vector.ipc.ReadChannel; +import org.apache.arrow.vector.ipc.message.MessageSerializer; import org.apache.arrow.vector.types.pojo.Schema; /** @@ -63,6 +66,43 @@ public DataFrame sql(String query) { return new DataFrame(dfHandle); } + /** + * Decode a DataFusion-Proto {@code LogicalPlanNode} and return a lazy {@link DataFrame}. The plan + * is not executed until {@link DataFrame#collect} is called. + * + *

The bytes must be a serialized {@code datafusion.LogicalPlanNode} (see {@code + * org.apache.datafusion.protobuf.LogicalPlanNode}). + * + * @throws RuntimeException if the bytes are not a valid {@code LogicalPlanNode} or if logical + * planning fails. + */ + public DataFrame fromProto(byte[] planBytes) { + if (nativeHandle == 0) { + throw new IllegalStateException("SessionContext is closed"); + } + long dfHandle = createDataFrameFromProto(nativeHandle, planBytes); + return new DataFrame(dfHandle); + } + + /** + * Return the Arrow {@link Schema} of a registered table. Transferred via Arrow IPC; no {@link + * org.apache.arrow.memory.BufferAllocator} is required because a schema carries no buffer data. + * + * @throws RuntimeException if {@code tableName} is not registered in this context. + */ + public Schema tableSchema(String tableName) { + if (nativeHandle == 0) { + throw new IllegalStateException("SessionContext is closed"); + } + byte[] ipcBytes = tableSchemaIpc(nativeHandle, tableName); + try { + return MessageSerializer.deserializeSchema( + new ReadChannel(Channels.newChannel(new ByteArrayInputStream(ipcBytes)))); + } catch (IOException e) { + throw new RuntimeException("Failed to deserialize IPC schema", e); + } + } + public void registerParquet(String name, String path) { registerParquet(name, path, new ParquetReadOptions()); } @@ -142,6 +182,10 @@ public void close() { private static native long createDataFrame(long handle, String sql); + private static native long createDataFrameFromProto(long handle, byte[] planBytes); + + private static native byte[] tableSchemaIpc(long handle, String tableName); + private static native void registerParquetWithOptions( long handle, String name, diff --git a/src/main/java/org/apache/datafusion/proto/SchemaConverter.java b/src/main/java/org/apache/datafusion/proto/SchemaConverter.java new file mode 100644 index 0000000..96d0d76 --- /dev/null +++ b/src/main/java/org/apache/datafusion/proto/SchemaConverter.java @@ -0,0 +1,212 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.datafusion.proto; + +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; + +import org.apache.arrow.vector.types.DateUnit; +import org.apache.arrow.vector.types.FloatingPointPrecision; +import org.apache.arrow.vector.types.pojo.ArrowType; +import org.apache.arrow.vector.types.pojo.Field; +import org.apache.arrow.vector.types.pojo.FieldType; +import org.apache.arrow.vector.types.pojo.Schema; + +import datafusion_common.DatafusionCommon; + +/** + * Convert between Arrow Java {@link Schema} and the {@code datafusion_common.Schema} protobuf shape + * used by DataFusion plan messages such as {@code ListingTableScanNode.schema}. + * + *

Supports the primitive Arrow types this project's tests exercise (Bool, signed/unsigned Int + * 8..64, Float32/64, Utf8, Utf8View, LargeUtf8, Date32, Decimal128). Anything else raises {@link + * UnsupportedOperationException} with a message naming the offending type. + */ +public final class SchemaConverter { + + private SchemaConverter() {} + + public static DatafusionCommon.Schema toProto(Schema arrow) { + DatafusionCommon.Schema.Builder builder = DatafusionCommon.Schema.newBuilder(); + for (Field f : arrow.getFields()) { + builder.addColumns(fieldToProto(f)); + } + if (arrow.getCustomMetadata() != null) { + builder.putAllMetadata(arrow.getCustomMetadata()); + } + return builder.build(); + } + + public static Schema fromProto(DatafusionCommon.Schema proto) { + List fields = new ArrayList<>(proto.getColumnsCount()); + for (DatafusionCommon.Field f : proto.getColumnsList()) { + fields.add(fieldFromProto(f)); + } + Map metadata = new LinkedHashMap<>(proto.getMetadataMap()); + return new Schema(fields, metadata); + } + + static DatafusionCommon.Field fieldToProto(Field f) { + DatafusionCommon.Field.Builder b = + DatafusionCommon.Field.newBuilder() + .setName(f.getName()) + .setArrowType(arrowTypeToProto(f.getType())) + .setNullable(f.isNullable()); + if (f.getMetadata() != null) { + b.putAllMetadata(f.getMetadata()); + } + return b.build(); + } + + static Field fieldFromProto(DatafusionCommon.Field f) { + ArrowType type = arrowTypeFromProto(f.getArrowType()); + FieldType ft = new FieldType(f.getNullable(), type, null, f.getMetadataMap()); + return new Field(f.getName(), ft, null); + } + + static DatafusionCommon.ArrowType arrowTypeToProto(ArrowType t) { + DatafusionCommon.EmptyMessage empty = DatafusionCommon.EmptyMessage.getDefaultInstance(); + DatafusionCommon.ArrowType.Builder b = DatafusionCommon.ArrowType.newBuilder(); + + if (t instanceof ArrowType.Bool) { + return b.setBOOL(empty).build(); + } + if (t instanceof ArrowType.Int) { + ArrowType.Int i = (ArrowType.Int) t; + if (i.getIsSigned()) { + switch (i.getBitWidth()) { + case 8: + return b.setINT8(empty).build(); + case 16: + return b.setINT16(empty).build(); + case 32: + return b.setINT32(empty).build(); + case 64: + return b.setINT64(empty).build(); + default: + throw new UnsupportedOperationException( + "Arrow type Int signed width " + + i.getBitWidth() + + " not yet supported by SchemaConverter"); + } + } else { + switch (i.getBitWidth()) { + case 8: + return b.setUINT8(empty).build(); + case 16: + return b.setUINT16(empty).build(); + case 32: + return b.setUINT32(empty).build(); + case 64: + return b.setUINT64(empty).build(); + default: + throw new UnsupportedOperationException( + "Arrow type Int unsigned width " + + i.getBitWidth() + + " not yet supported by SchemaConverter"); + } + } + } + if (t instanceof ArrowType.FloatingPoint) { + FloatingPointPrecision p = ((ArrowType.FloatingPoint) t).getPrecision(); + if (p == FloatingPointPrecision.SINGLE) return b.setFLOAT32(empty).build(); + if (p == FloatingPointPrecision.DOUBLE) return b.setFLOAT64(empty).build(); + throw new UnsupportedOperationException( + "Arrow type FloatingPoint " + p + " not yet supported by SchemaConverter"); + } + if (t instanceof ArrowType.Utf8) { + return b.setUTF8(empty).build(); + } + if (t instanceof ArrowType.Utf8View) { + return b.setUTF8VIEW(empty).build(); + } + if (t instanceof ArrowType.LargeUtf8) { + return b.setLARGEUTF8(empty).build(); + } + if (t instanceof ArrowType.Date) { + DateUnit u = ((ArrowType.Date) t).getUnit(); + if (u == DateUnit.DAY) return b.setDATE32(empty).build(); + throw new UnsupportedOperationException( + "Arrow type Date " + u + " not yet supported by SchemaConverter"); + } + if (t instanceof ArrowType.Decimal) { + ArrowType.Decimal d = (ArrowType.Decimal) t; + if (d.getBitWidth() != 128) { + throw new UnsupportedOperationException( + "Arrow type Decimal bit width " + + d.getBitWidth() + + " not yet supported by SchemaConverter"); + } + return b.setDECIMAL128( + DatafusionCommon.Decimal128Type.newBuilder() + .setPrecision(d.getPrecision()) + .setScale(d.getScale()) + .build()) + .build(); + } + throw new UnsupportedOperationException( + "Arrow type " + t.getClass().getSimpleName() + " not yet supported by SchemaConverter"); + } + + static ArrowType arrowTypeFromProto(DatafusionCommon.ArrowType p) { + switch (p.getArrowTypeEnumCase()) { + case BOOL: + return ArrowType.Bool.INSTANCE; + case INT8: + return new ArrowType.Int(8, true); + case INT16: + return new ArrowType.Int(16, true); + case INT32: + return new ArrowType.Int(32, true); + case INT64: + return new ArrowType.Int(64, true); + case UINT8: + return new ArrowType.Int(8, false); + case UINT16: + return new ArrowType.Int(16, false); + case UINT32: + return new ArrowType.Int(32, false); + case UINT64: + return new ArrowType.Int(64, false); + case FLOAT32: + return new ArrowType.FloatingPoint(FloatingPointPrecision.SINGLE); + case FLOAT64: + return new ArrowType.FloatingPoint(FloatingPointPrecision.DOUBLE); + case UTF8: + return ArrowType.Utf8.INSTANCE; + case UTF8_VIEW: + return ArrowType.Utf8View.INSTANCE; + case LARGE_UTF8: + return ArrowType.LargeUtf8.INSTANCE; + case DATE32: + return new ArrowType.Date(DateUnit.DAY); + case DECIMAL128: + DatafusionCommon.Decimal128Type d = p.getDECIMAL128(); + return new ArrowType.Decimal(d.getPrecision(), d.getScale(), 128); + default: + throw new UnsupportedOperationException( + "datafusion_common.ArrowType " + + p.getArrowTypeEnumCase() + + " not yet supported by SchemaConverter"); + } + } +} diff --git a/src/test/java/org/apache/datafusion/proto/SchemaConverterTest.java b/src/test/java/org/apache/datafusion/proto/SchemaConverterTest.java new file mode 100644 index 0000000..dd4cd23 --- /dev/null +++ b/src/test/java/org/apache/datafusion/proto/SchemaConverterTest.java @@ -0,0 +1,107 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.datafusion.proto; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.util.Arrays; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import org.apache.arrow.vector.types.DateUnit; +import org.apache.arrow.vector.types.FloatingPointPrecision; +import org.apache.arrow.vector.types.pojo.ArrowType; +import org.apache.arrow.vector.types.pojo.Field; +import org.apache.arrow.vector.types.pojo.FieldType; +import org.apache.arrow.vector.types.pojo.Schema; +import org.junit.jupiter.api.Test; + +import datafusion_common.DatafusionCommon; + +class SchemaConverterTest { + + @Test + void roundTripsPrimitivesAndMetadata() { + Map schemaMeta = new HashMap<>(); + schemaMeta.put("origin", "test"); + Map fieldMeta = new HashMap<>(); + fieldMeta.put("ns", "demo"); + + List fields = + Arrays.asList( + new Field("a_bool", FieldType.nullable(ArrowType.Bool.INSTANCE), null), + new Field("a_i32", FieldType.notNullable(new ArrowType.Int(32, true)), null), + new Field("a_i64", FieldType.nullable(new ArrowType.Int(64, true)), null), + new Field("a_u32", FieldType.nullable(new ArrowType.Int(32, false)), null), + new Field( + "a_f64", + FieldType.nullable(new ArrowType.FloatingPoint(FloatingPointPrecision.DOUBLE)), + null), + new Field("a_str", FieldType.nullable(ArrowType.Utf8.INSTANCE), null), + new Field("a_date", FieldType.nullable(new ArrowType.Date(DateUnit.DAY)), null), + new Field( + "a_with_meta", + new FieldType(true, new ArrowType.Int(8, true), null, fieldMeta), + null)); + + Schema original = new Schema(fields, schemaMeta); + + DatafusionCommon.Schema proto = SchemaConverter.toProto(original); + Schema roundTripped = SchemaConverter.fromProto(proto); + + assertEquals(original, roundTripped); + } + + @Test + void decimalPreservesPrecisionAndScale() { + Schema original = + new Schema( + List.of( + new Field("amount", FieldType.nullable(new ArrowType.Decimal(18, 5, 128)), null))); + + DatafusionCommon.Schema proto = SchemaConverter.toProto(original); + Schema roundTripped = SchemaConverter.fromProto(proto); + + assertEquals(original, roundTripped); + ArrowType.Decimal d = (ArrowType.Decimal) roundTripped.getFields().get(0).getType(); + assertEquals(18, d.getPrecision()); + assertEquals(5, d.getScale()); + assertEquals(128, d.getBitWidth()); + } + + @Test + void unsupportedTypeRaisesUnsupportedOperationException() { + Field listField = + new Field( + "nested", + FieldType.nullable(new ArrowType.List()), + List.of(new Field("item", FieldType.nullable(new ArrowType.Int(32, true)), null))); + Schema original = new Schema(List.of(listField)); + + UnsupportedOperationException ex = + assertThrows(UnsupportedOperationException.class, () -> SchemaConverter.toProto(original)); + assertTrue( + ex.getMessage().contains("List"), + "exception message should name the unsupported type, was: " + ex.getMessage()); + } +} diff --git a/src/test/java/org/apache/datafusion/proto/SessionContextProtoTest.java b/src/test/java/org/apache/datafusion/proto/SessionContextProtoTest.java new file mode 100644 index 0000000..7bb65e6 --- /dev/null +++ b/src/test/java/org/apache/datafusion/proto/SessionContextProtoTest.java @@ -0,0 +1,188 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.datafusion.proto; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.nio.file.Files; +import java.nio.file.Path; + +import org.apache.arrow.memory.BufferAllocator; +import org.apache.arrow.memory.RootAllocator; +import org.apache.arrow.vector.BigIntVector; +import org.apache.arrow.vector.IntVector; +import org.apache.arrow.vector.VectorSchemaRoot; +import org.apache.arrow.vector.ipc.ArrowReader; +import org.apache.arrow.vector.types.DateUnit; +import org.apache.arrow.vector.types.pojo.ArrowType; +import org.apache.arrow.vector.types.pojo.Schema; +import org.apache.datafusion.DataFrame; +import org.apache.datafusion.SessionContext; +import org.apache.datafusion.protobuf.BareTableReference; +import org.apache.datafusion.protobuf.EmptyRelationNode; +import org.apache.datafusion.protobuf.ListingTableScanNode; +import org.apache.datafusion.protobuf.LogicalExprNode; +import org.apache.datafusion.protobuf.LogicalPlanNode; +import org.apache.datafusion.protobuf.ProjectionColumns; +import org.apache.datafusion.protobuf.ProjectionNode; +import org.apache.datafusion.protobuf.SortExprNode; +import org.apache.datafusion.protobuf.SortNode; +import org.apache.datafusion.protobuf.TableReference; +import org.junit.jupiter.api.Assumptions; +import org.junit.jupiter.api.Test; + +import datafusion_common.DatafusionCommon; + +class SessionContextProtoTest { + + @Test + void fromProtoExecutesProjectionOverEmptyRelation() throws Exception { + LogicalPlanNode plan = + LogicalPlanNode.newBuilder() + .setProjection( + ProjectionNode.newBuilder() + .setInput( + LogicalPlanNode.newBuilder() + .setEmptyRelation( + EmptyRelationNode.newBuilder().setProduceOneRow(true).build()) + .build()) + .addExpr( + LogicalExprNode.newBuilder() + .setLiteral( + DatafusionCommon.ScalarValue.newBuilder().setInt32Value(1).build()) + .build()) + .build()) + .build(); + + try (BufferAllocator allocator = new RootAllocator(); + SessionContext ctx = new SessionContext(); + DataFrame df = ctx.fromProto(plan.toByteArray()); + ArrowReader reader = df.collect(allocator)) { + assertTrue(reader.loadNextBatch()); + VectorSchemaRoot root = reader.getVectorSchemaRoot(); + assertEquals(1, root.getRowCount()); + IntVector col = (IntVector) root.getVector(0); + assertEquals(1, col.get(0)); + } + } + + @Test + void tableSchemaReturnsExpectedLineitemFields() { + Path lineitem = Path.of("tpch-data/sf1/lineitem.parquet"); + Assumptions.assumeTrue( + Files.exists(lineitem), "TPC-H SF1 data not found; run `make tpch-data` first"); + + try (SessionContext ctx = new SessionContext()) { + ctx.registerParquet("lineitem", lineitem.toAbsolutePath().toString()); + Schema schema = ctx.tableSchema("lineitem"); + + assertEquals(16, schema.getFields().size()); + assertEquals("l_orderkey", schema.getFields().get(0).getName()); + assertTrue(schema.getFields().get(0).getType() instanceof ArrowType.Int); + + // l_extendedprice = Decimal128(15, 2) + ArrowType.Decimal price = (ArrowType.Decimal) schema.findField("l_extendedprice").getType(); + assertEquals(15, price.getPrecision()); + assertEquals(2, price.getScale()); + + // l_shipdate = Date(DAY) + ArrowType.Date ship = (ArrowType.Date) schema.findField("l_shipdate").getType(); + assertEquals(DateUnit.DAY, ship.getUnit()); + } + } + + @Test + void fromProtoListingScanMatchesSql() throws Exception { + Path lineitem = Path.of("tpch-data/sf1/lineitem.parquet"); + Assumptions.assumeTrue( + Files.exists(lineitem), "TPC-H SF1 data not found; run `make tpch-data` first"); + String absPath = lineitem.toAbsolutePath().toString(); + + try (BufferAllocator allocator = new RootAllocator(); + SessionContext ctx = new SessionContext()) { + ctx.registerParquet("lineitem", absPath); + Schema arrow = ctx.tableSchema("lineitem"); + DatafusionCommon.Schema schemaProto = SchemaConverter.toProto(arrow); + + LogicalExprNode orderKeyExpr = + LogicalExprNode.newBuilder() + .setColumn(DatafusionCommon.Column.newBuilder().setName("l_orderkey").build()) + .build(); + + LogicalPlanNode plan = + LogicalPlanNode.newBuilder() + .setSort( + SortNode.newBuilder() + .setInput( + LogicalPlanNode.newBuilder() + .setListingScan( + ListingTableScanNode.newBuilder() + .setTableName( + TableReference.newBuilder() + .setBare( + BareTableReference.newBuilder() + .setTable("lineitem") + .build()) + .build()) + .addPaths(absPath) + .setFileExtension(".parquet") + .setSchema(schemaProto) + .setProjection( + ProjectionColumns.newBuilder() + .addColumns("l_orderkey") + .build()) + .setParquet( + DatafusionCommon.ParquetFormat.getDefaultInstance()) + .setTargetPartitions(1) + .build()) + .build()) + .addExpr( + SortExprNode.newBuilder() + .setExpr(orderKeyExpr) + .setAsc(true) + .setNullsFirst(false) + .build()) + .setFetch(1) + .build()) + .build(); + + long protoValue; + try (DataFrame df = ctx.fromProto(plan.toByteArray()); + ArrowReader reader = df.collect(allocator)) { + assertTrue(reader.loadNextBatch()); + VectorSchemaRoot root = reader.getVectorSchemaRoot(); + assertEquals(1, root.getRowCount()); + protoValue = ((BigIntVector) root.getVector(0)).get(0); + } + + long sqlValue; + try (DataFrame df = ctx.sql("SELECT l_orderkey FROM lineitem ORDER BY l_orderkey LIMIT 1"); + ArrowReader reader = df.collect(allocator)) { + assertTrue(reader.loadNextBatch()); + VectorSchemaRoot root = reader.getVectorSchemaRoot(); + assertEquals(1, root.getRowCount()); + sqlValue = ((BigIntVector) root.getVector(0)).get(0); + } + + assertEquals(sqlValue, protoValue); + } + } +} From 82ef7ad691c3a10c0d5e6ec7620a115e1ccb0220 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Wed, 13 May 2026 00:40:11 -0600 Subject: [PATCH 14/14] docs: replace roadmap with project-status checklist Rewrite the README around a feature checklist covering the three query interfaces (SQL, limited DataFrame, DataFusion-Proto logical plans), data sources, result materialization, and not-yet items. Move build prerequisites, build/test commands, and the TPC-H test-data section from README into CONTRIBUTING.md so the README focuses on using the library rather than building it. --- CONTRIBUTING.md | 34 +++++++++++++++ README.md | 108 ++++++++++++++---------------------------------- 2 files changed, 65 insertions(+), 77 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 302b6b3..d9e93ac 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -41,6 +41,40 @@ to fetch the DataFusion protobuf schemas (see *Updating the DataFusion / protobuf schema version* below). Subsequent builds are offline — the `download-maven-plugin` cache under `~/.m2/repository/.cache/` satisfies them. +## Build prerequisites + +- JDK 17 or newer. +- Rust toolchain (stable, installed via [rustup]). +- [`tpchgen-cli`] — only needed to generate test data for the Parquet + integration test (`cargo install tpchgen-cli`). + +Maven is bundled via the `./mvnw` wrapper; no separate Maven install required. + +[rustup]: https://rustup.rs/ +[`tpchgen-cli`]: https://github.com/clflushopt/tpchgen-rs + +## Build and test + + make test + +This builds the native Rust crate and runs the JUnit tests. The steps can be +run individually: + + cd native && cargo build + ./mvnw test + +The native library must be built before running JVM tests. + +## Test data + +The Parquet integration test reads TPC-H SF1 data (~345 MB across 8 tables in +Snappy-compressed Parquet). Generate it once with: + + make tpch-data + +Tests that need this data skip cleanly if it is missing. `make clean` does +**not** remove `tpch-data/` — delete it manually to reclaim the disk space. + ## Code style - Java: run `./mvnw spotless:apply` before committing. CI fails the build if diff --git a/README.md b/README.md index f613008..21637cd 100644 --- a/README.md +++ b/README.md @@ -1,17 +1,13 @@ # Apache DataFusion Java -Java bindings for [Apache DataFusion], the Rust-based query engine. SQL queries -run in native code and results are returned to the JVM as [Apache Arrow] record -batches via the Arrow C Data Interface — no per-row JNI calls, no row-by-row -copies. +Java bindings for [Apache DataFusion]. Queries run in native Rust and results +return to the JVM as [Apache Arrow] batches via the Arrow C Data Interface. [Apache DataFusion]: https://datafusion.apache.org/ [Apache Arrow]: https://arrow.apache.org/ -> **Project status: early development.** This is a brand-new project. The API -> is small, will change without notice, and there is no published release. Do -> not depend on it from production code yet. Bug reports, design feedback, and -> contributions are very welcome. +> Early development: no releases yet, API will change. Bug reports and +> contributions welcome. ## Quickstart @@ -32,97 +28,55 @@ try (var allocator = new RootAllocator(); ArrowReader reader = df.collect(allocator)) { while (reader.loadNextBatch()) { var batch = reader.getVectorSchemaRoot(); - // ... consume batch ... + // ... } } } ``` -The current public surface mirrors a small slice of the Rust DataFusion API: +`SessionContext` and `DataFrame` are `AutoCloseable` and not thread-safe. -- `SessionContext.sql(String query)` — parse and plan a SQL query, returning a - lazy `DataFrame`. No execution happens yet. -- `DataFrame.collect(BufferAllocator allocator)` — execute the plan and return - the result batches as an `ArrowReader`. Consumes the DataFrame; the caller - closes the reader, and the allocator must outlive it. -- `SessionContext.registerParquet(String name, String path)` — register a local - Parquet file as a SQL table. +## Project status -Both `SessionContext` and `DataFrame` are `AutoCloseable` and **not -thread-safe**. +Query interfaces: -## Prerequisites +- [x] SQL: `SessionContext.sql(String)` +- [x] DataFrame: `select`, `filter` (other transformations TBD) +- [x] DataFusion-Proto `LogicalPlanNode`: `SessionContext.fromProto(byte[])`. + The `datafusion-proto` Java classes are generated by the build. -- JDK 17 or newer -- Rust toolchain (stable, installed via [rustup]) -- [`tpchgen-cli`] — only needed to generate test data for the Parquet - integration test (`cargo install tpchgen-cli`) +Data sources: -Maven is bundled via the `./mvnw` wrapper; no separate Maven install required. +- [x] Parquet via `registerParquet` / `readParquet`, with `ParquetReadOptions` +- [ ] CSV, JSON, Avro +- [ ] Custom catalog and table providers -[rustup]: https://rustup.rs/ -[`tpchgen-cli`]: https://github.com/clflushopt/tpchgen-rs +Results: -## Build & test +- [x] `DataFrame.collect(allocator)` — Arrow C Data Interface stream +- [x] `DataFrame.count()`, `show()`, `show(int)` +- [x] `SessionContext.tableSchema(String)` - make test +Not yet: -This builds the native Rust crate and runs the JUnit tests. The steps can be -run individually: +- [ ] `SessionConfig` / `RuntimeEnv` knobs +- [ ] Java UDFs +- [ ] `write_*` outputs - cd native && cargo build - ./mvnw test +## Requirements -The native library must be built before running JVM tests. +JDK 17+. Building from source: see [CONTRIBUTING.md](CONTRIBUTING.md). -## Test data - -The Parquet integration test reads TPC-H SF1 data (~345 MB across 8 tables in -Snappy-compressed Parquet). Generate it once with: - - make tpch-data - -Tests that need this data skip cleanly if it is missing. `make clean` does -**not** remove `tpch-data/` — delete it manually to reclaim the disk space. - -## Repository layout +## Layout - `src/` — Java sources and tests -- `native/` — Rust crate that exposes DataFusion over JNI and the Arrow C Data - Interface - -## Roadmap - -Near-term priorities, roughly in order: - -- **Session configuration.** Expose `SessionConfig` and `RuntimeEnv` settings - (target partitions, batch size, memory pool, default catalog, …) so callers - can tune execution from the JVM. -- **Full `SessionContext` and `DataFrame` APIs.** Expand beyond `sql` and - `registerParquet` to mirror the Rust API: table registration variants, - `read_*` / `write_*` entry points, and DataFrame transformations such as - `select`, `filter`, `join`, `aggregate`, `sort`, plus result-materialization - variants (`show`, `count`, streaming collection). -- **JVM-side plan construction via Protobuf.** Build DataFusion logical and - physical plans on the JVM using the existing DataFusion Protobuf - representation, then ship them to the native side for execution. This lets - plans assembled by other JVM tools (Spark, Flink, Beam, custom planners) run - on DataFusion without having to go through SQL. -- **Java-defined expressions.** Allow user expressions / UDFs to be implemented - in Java and invoked from the native plan, operating on Apache Arrow vectors - (Java side) so evaluation stays vectorized end-to-end and avoids row-by-row - JNI crossings. - -These are intentionally large items — design discussion via GitHub issues -before implementation is welcome. +- `native/` — Rust crate (JNI + Arrow C Data Interface) ## Contributing -This project follows the Apache DataFusion contribution model. Issues and pull -requests are welcome — please open a GitHub issue to discuss any significant -change before sending a PR. +Open an issue to discuss non-trivial changes before sending a PR. +See [CONTRIBUTING.md](CONTRIBUTING.md). ## License -Licensed under the [Apache License, Version 2.0](LICENSE.txt). See -[NOTICE.txt](NOTICE.txt) for required attributions. +Apache License 2.0. See [LICENSE.txt](LICENSE.txt) and [NOTICE.txt](NOTICE.txt).