Conversation
## Which issue does this PR close? - Closes apache#21862. ## Rationale for this change Similar to other recent changes, we can use the bulk-NULL string builders to avoid per-row NULL handling overhead. Benchmarks: - scalar/scalar_utf8: 227.4ns → 228.6ns (+0.53%) - scalar/scalar_utf8view: 227.9ns → 229.2ns (+0.57%) - size=1024 str_len=128/array_utf8: 276.5µs → 277.7µs (+0.43%) - size=1024 str_len=128/array_utf8view: 223.3µs → 222.4µs (−0.40%) - size=1024 str_len=16/array_utf8: 37.5µs → 37.4µs (−0.27%) - size=1024 str_len=16/array_utf8view: 41.9µs → 39.3µs (−6.21%) - size=4096 str_len=128/array_utf8: 1115.4µs → 1109.2µs (−0.56%) - size=4096 str_len=128/array_utf8view: 900.2µs → 895.2µs (−0.56%) - size=4096 str_len=16/array_utf8: 148.3µs → 148.4µs (+0.07%) - size=4096 str_len=16/array_utf8view: 164.2µs → 153.8µs (−6.33%) - size=8192 str_len=128/array_utf8: 2.2ms → 2.2ms (0.00%) - size=8192 str_len=128/array_utf8view: 1811.2µs → 1819.2µs (+0.44%) - size=8192 str_len=16/array_utf8: 299.0µs → 299.3µs (+0.10%) - size=8192 str_len=16/array_utf8view: 330.6µs → 310.2µs (−6.17%) - unicode size=1024/array_utf8: 772.2µs → 769.2µs (−0.39%) - unicode size=1024/array_utf8view: 777.8µs → 779.2µs (+0.18%) - unicode size=4096/array_utf8: 3.1ms → 3.1ms (0.00%) - unicode size=4096/array_utf8view: 3.1ms → 3.1ms (0.00%) - unicode size=8192/array_utf8: 6.2ms → 6.1ms (−1.61%) - unicode size=8192/array_utf8view: 6.2ms → 6.2ms (0.00%) ## What changes are included in this PR? * Switch to bulk-NULL string builders * Optimize `StringViewArrayBuilder::append_value()` by inlining the view construction for > 12 byte strings. Empirically this improved performance, probably because `make_view` can't be inlined. This should also help other PRs in this series. ## Are these changes tested? Yes, covered by existing tests. ## Are there any user-facing changes? No.
## Which issue does this PR close? - Closes apache#21574. ## Rationale for this change Previously, the CLI hint was being highlighted directly, which was causing an assertion to fail in `rustyline`. ## What changes are included in this PR? - Use `highlight_hint` instead of manually highlighting the text. ## Are these changes tested? Manually. ## Are there any user-facing changes? No.
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes apache#21279 - Closes apache#21281 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Keep dependencies up to date. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> We used to use `Digest` from `blake2`, which was a common dependency used by `md5` and `sha2`; however `blake2` doesn't have a `0.11.0` release so we were blocked because of incompatible dependencies when trying to upgrade `sha2` or `md5`. Fix code to use their own `Digest` (e.g. `md5::Digest`, `sha2::Digest`) instead of relying on `blake2::Digest` which should prevent such issues from occurring again and allows us to bump their versions independently. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Existing tests. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> No. <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->
## Which issue does this PR close? - None. This is a small documentation sync update. ## Rationale for this change The CLI help output shown in the user guide had drifted from the current `datafusion-cli --help` output, which makes the documentation misleading for users trying to discover supported flags. ## What changes are included in this PR? - Update the `datafusion-cli --help` snippet in `docs/source/user-guide/cli/usage.md` - Remove outdated options and defaults - Add newer flags and current help text formatting ## Are these changes tested? - Not with automated tests. This is a documentation-only change that updates the checked-in help output. ## Are there any user-facing changes? - Yes. The user guide now matches the current CLI usage output more closely. Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
## Which issue does this PR close? - Follow on to apache#21240 ## Rationale for this change While reviewing apache#21240 I found that the existing doc comments describing the relationship between ExecutionProps and TaskContext were vague ## What changes are included in this PR? - Update the doc comments on ExecutionProps and TaskContext - Clarify why the two structures remain separate ## Are these changes tested? Not with automated tests. This is a comment-only change. ## Are there any user-facing changes? Docs only
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes apache#21747 ## Rationale for this change datafusion did not have a CI check for broken links in markdown content, docs workflows build and deploy docs, and dev checks formatting and spelling, but none of them validate link targets. This pr adds a dedicated link check for internal markdown links so broken references fail early in PRs. I kept the scope internal-only to avoid flaky CI failures from external websites and rate limits. Rust doc comments remain covered by the existing rustdoc CI job. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? - Added a new Dev workflow job, **Check Markdown Links**, in `dev.yml`. - Added `LYCHEE_VERSION` pin in `tool_versions.sh`. - Added `markdown_link_check.sh` to run lychee on the selected markdown paths. - Added `lychee.toml` with internal-link policy and exclusions. - Added check markdown links to required status checks in `.asf.yaml`. - Updated contributor testing docs with the new local command and scope note. - Fixed internal markdown links that failed under the new check in: - `roadmap.md` - `49.0.0.md` - `overview.md` - `dataframe.md` - `format_options.md` <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? Yes, - `python3 ci/scripts/check_asf_yaml_status_checks.py` passed. - `bash -n ci/scripts/markdown_link_check.sh` passed. - `bash ci/scripts/markdown_link_check.sh` passed with 0 errors. - `cargo fmt --all --check` passed. OK: All 5 required_status_checks match existing GitHub Actions jobs. 🔍 12824 Total (in 0s) ✅ 490 OK 🚫 0 Errors 👻 12334 Excluded <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? No, There is one contributor-facing CI change: PRs now fail when internal markdown links break in the checked markdown files. <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Oleks V <comphead@users.noreply.github.com>
## Which issue does this PR close? - Closes apache#21846. ## Rationale for this change Optimize `chr` by avoiding per-row NULL bitmap maintenance, and also split the hot loop to avoid taking a branch when no NULL bitmap is given. Benchmarks: ``` chr/array: 3.8768 µs → 3.1548 µs, −18.57% (`p < 0.05`) ``` ## What changes are included in this PR? * Optimize `chr` to reduce NULL-handling overhead ## Are these changes tested? Yes. ## Are there any user-facing changes? No. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
…ache#21725) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes apache#16894 ## What changes are included in this PR? - User-visible struct-UNNEST columns no longer expose `__unnest_placeholder`. - Internal Unnest planning still uses placeholders. - This adds a final Projection layer in explain plans wherever struct UNNEST output is published. <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? Yes. <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? Yes. The aliases of unnested columns will be changed. <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
## Which issue does this PR close? - Part of apache#17964. ## Rationale for this change SparkAvg's AvgGroupsAccumulator doesn't implement supports_convert_to_state (defaults to false), which prevents the skip-partial-aggregation optimization from kicking in for queries that use Spark's avg(). I ran into this while benchmarking a Spark Connect engine built on DataFusion. On TPC-H q17 at SF10, the partial aggregate for avg(l_quantity) grouped by l_partkey (~2M groups out of 60M rows) was not triggering skip-aggregation: | Metric | Without convert_to_state | With convert_to_state | |--------|-------------------------|-----------------------| | Partial aggregate memory | 923 MB | 40 MB | | Partial aggregate elapsed | 4.75s | 109ms | The skip-aggregation probe (apache#11627) detects when a partial aggregate isn't reducing cardinality and falls back to passing rows through as state directly. This needs convert_to_state so the accumulator can produce [sum, count] state arrays from raw input. The built-in Avg already has this (apache#11734), but it wasn't carried over when SparkAvg was migrated from Comet in apache#17871. ## What changes are included in this PR? Adds convert_to_state() and supports_convert_to_state() to AvgGroupsAccumulator in datafusion-spark. Follows the same approach as the built-in Avg, adapted for SparkAvg's differences: - State order is [sum, count] (vs [count, sum] in the built-in) - Count type is Int64 (vs UInt64 in the built-in) - Null handling uses NullBuffer::union directly instead of pulling in datafusion-functions-aggregate-common as a dep Also cleaned up the fully-qualified arrow::array::BooleanArray references in update_batch / merge_batch since adding BooleanArray to the import block triggered the unused_qualifications lint. ## Are these changes tested? Yes, unit tests covering basic conversion, null propagation, filter handling, and a roundtrip through merge_batch to verify the converted state produces correct results end-to-end. ## Are there any user-facing changes? No. Queries using avg() through the Spark function registry will automatically benefit from skip-partial-aggregation on high-cardinality groupings. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
…pache#21240) ## Which issue does this PR close? - Closes apache#3781. - Closes apache#18181. ## Rationale for this change Previously, DataFusion evaluated uncorrelated scalar subqueries by transforming them into joins. This has three shortcomings: 1. Scalar subqueries that return > 1 row were allowed, producing incorrect query results. Such queries should instead result in a runtime error. 2. Performance. Evaluating scalar subqueries as a join requires going through the join machinery. More importantly, it means that UDFs that have specialized handling of scalar inputs cannot use those code paths for scalar subqueries, which often results in significantly slower query execution (e.g., apache#18181). It also makes filter pushdown for scalar subquery filters more difficult (apache#21324) 3. Uncorrelated scalar subqueries previously did not work in `ORDER BY` or `JOIN ON`, or as arguments to an aggregate function. Those cases are now supported. This PR introduces physical execution of uncorrelated scalar subqueries: * Uncorrelated subqueries are left in the plan by the optimizer, not rewritten into joins * The physical planner collects uncorrelated scalar subqueries and plans them recursively (supporting nested subqueries). We add a `ScalarSubqueryExec` plan node to the top of any physical plan with uncorrelated subqueries: it has N+1 children, N subqueries and its "main" input, which is the rest of the query plan. The subquery expression in the parent plan is replaced with a `ScalarSubqueryExpr`. * `ScalarSubqueryExec` manages the execution of the subqueries. Subquery evaluation is done in parallel (for a given query level), but at present it happens strictly before evaluation of the parent query. This might be improved in the future (apache#21591). * `ScalarSubqueryExpr` reads its value from a shared slot that `ScalarSubqueryExec` populates when the subquery finishes; the physical planner assigns each subquery its slot index via `ExecutionProps`. This architecture makes it easy to avoid the shortcomings described above. Performance seems roughly unchanged (benchmarks added in this PR), but in situations like apache#18181, we can now leverage scalar fast-paths; in the case of apache#18181 specifically, this improves performance from ~800 ms to ~30 ms. ## What changes are included in this PR? * Modify subquery rewriter to not transform subqueries -> joins * Collect and plan uncorrelated scalar subqueries in the physical planner, and wire up `ScalarSubqueryExpr` * Support for subqueries in physical plan serialization/deserialization using `PhysicalProtoConverterExtension` to wire up `ScalarSubqueryExpr` correctly * Support for subqueries in logical plan serialization/deserialization * Add various SLT tests and update expected plan shapes for some tests ## Are these changes tested? Yes. New SLT coverage for cardinality errors, `ORDER BY` / `JOIN ON` / aggregate-arg contexts, nested uncorrelated subqueries, duplicate-subquery deduplication, and partition-pruning filters; new roundtrip tests for logical and physical plan serialization. ## Are there any user-facing changes? SQL: * Uncorrelated scalar subqueries that return more than one row now result in a runtime error, instead of silently producing incorrect results. * Uncorrelated scalar subqueries now work in `ORDER BY`, `JOIN ON`, and as aggregate function arguments. Rust APIs: * In `datafusion-proto`, breaking changes to `Serializeable::from_bytes_with_registry` (renamed to `from_bytes_with_ctx`), `parse_expr` / `parse_sorts` / `parse_exprs`, and the `PhysicalProtoConverterExtension` trait. Plan shape: * `LogicalPlan::Subquery` nodes will now be preserved in the logical plan * Physical plans can now contain `ScalarSubqueryExec` plan node and `ScalarSubqueryExpr` expressions The wire format has also changed to include scalar subqueries. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
…#21383) ## Which issue does this PR close? - Refers to apache#17899 ## Rationale for this change A performance improvement of `first_last.rs` (up to 36%), based on the original author's observations. UPD: rebased on yesterday's refactoring in apache#17899 - сс @UBarney @alamb ## What changes are included in this PR? - Optimise memory allocation in `take_state` - no need to copy vectors and buffers - Optimise extracting single elements - Pre-compute common data for sorting ~~- Rename structs and functions as recommended in TODOs - a majority of changes in this PR~~ - Add benchmark. It's pretty complicated to test aggregates with grouping, since many operations are stateful, so I introduced end-to-end `evaluate` test (to actually test taking state) and `convert_to_state` (as in other benches) ## Are these changes tested? - Existing unit and integration tests - A new bench has a meaningful result Improvements: up to 36% Raw bench result: <details> first_value convert_to_state nulls=0%, filter=false time: [98.086 µs 99.014 µs 100.25 µs] change: [−15.400% −13.532% −11.559%] (p = 0.00 < 0.05) Performance has improved. Found 17 outliers among 100 measurements (17.00%) 4 (4.00%) low mild 3 (3.00%) high mild 10 (10.00%) high severe Benchmarking first_value evaluate_bench nulls=0%, filter=false, first(2): Collecting 100 samples in estimated 7.1575 s (10k ite first_value evaluate_bench nulls=0%, filter=false, first(2) time: [54.938 µs 55.482 µs 56.060 µs] change: [−38.163% −36.295% −34.463%] (p = 0.00 < 0.05) Performance has improved. Found 9 outliers among 100 measurements (9.00%) 5 (5.00%) high mild 4 (4.00%) high severe Benchmarking first_value evaluate_bench nulls=0%, filter=false, all: Collecting 100 samples in estimated 7.0741 s (10k iteratio first_value evaluate_bench nulls=0%, filter=false, all time: [50.624 µs 51.092 µs 51.612 µs] change: [−17.955% −16.332% −14.593%] (p = 0.00 < 0.05) Performance has improved. Found 9 outliers among 100 measurements (9.00%) 5 (5.00%) high mild 4 (4.00%) high severe first_value convert_to_state nulls=0%, filter=true time: [2.0647 µs 2.0881 µs 2.1148 µs] change: [−7.6314% −6.2174% −4.8530%] (p = 0.00 < 0.05) Performance has improved. Found 8 outliers among 100 measurements (8.00%) 2 (2.00%) high mild 6 (6.00%) high severe Benchmarking first_value evaluate_bench nulls=0%, filter=true, first(2): Collecting 100 samples in estimated 9.6354 s (10k iter first_value evaluate_bench nulls=0%, filter=true, first(2) time: [54.708 µs 55.240 µs 55.805 µs] change: [−27.249% −24.755% −22.195%] (p = 0.00 < 0.05) Performance has improved. Found 5 outliers among 100 measurements (5.00%) 3 (3.00%) high mild 2 (2.00%) high severe Benchmarking first_value evaluate_bench nulls=0%, filter=true, all: Collecting 100 samples in estimated 9.6394 s (10k iteration first_value evaluate_bench nulls=0%, filter=true, all time: [50.540 µs 50.963 µs 51.424 µs] change: [−6.8346% −3.9146% −0.8864%] (p = 0.01 < 0.05) Change within noise threshold. Found 4 outliers among 100 measurements (4.00%) 1 (1.00%) high mild 3 (3.00%) high severe Benchmarking first_value convert_to_state nulls=90%, filter=false: Collecting 100 samples in estimated 5.0440 s (50k iterations first_value convert_to_state nulls=90%, filter=false time: [98.054 µs 98.996 µs 100.10 µs] change: [−3.4798% −2.2129% −1.0173%] (p = 0.00 < 0.05) Performance has improved. Found 19 outliers among 100 measurements (19.00%) 1 (1.00%) low severe 8 (8.00%) low mild 2 (2.00%) high mild 8 (8.00%) high severe Benchmarking first_value evaluate_bench nulls=90%, filter=false, first(2): Collecting 100 samples in estimated 8.5385 s (10k it first_value evaluate_bench nulls=90%, filter=false, first(2) time: [53.780 µs 54.673 µs 55.639 µs] change: [−17.702% −15.978% −14.173%] (p = 0.00 < 0.05) Performance has improved. Found 8 outliers among 100 measurements (8.00%) 1 (1.00%) low mild 5 (5.00%) high mild 2 (2.00%) high severe Benchmarking first_value evaluate_bench nulls=90%, filter=false, all: Collecting 100 samples in estimated 8.2692 s (10k iterati first_value evaluate_bench nulls=90%, filter=false, all time: [49.851 µs 50.289 µs 50.755 µs] change: [−4.8554% −3.1896% −1.3951%] (p = 0.00 < 0.05) Performance has improved. Found 8 outliers among 100 measurements (8.00%) 3 (3.00%) low severe 2 (2.00%) high mild 3 (3.00%) high severe Benchmarking first_value convert_to_state nulls=90%, filter=true: Collecting 100 samples in estimated 5.0077 s (2.4M iterations first_value convert_to_state nulls=90%, filter=true time: [2.0339 µs 2.0465 µs 2.0603 µs] change: [−1.8037% −0.7068% +0.3821%] (p = 0.22 > 0.05) No change in performance detected. Found 8 outliers among 100 measurements (8.00%) 2 (2.00%) low mild 3 (3.00%) high mild 3 (3.00%) high severe Benchmarking first_value evaluate_bench nulls=90%, filter=true, first(2): Collecting 100 samples in estimated 9.7925 s (10k ite first_value evaluate_bench nulls=90%, filter=true, first(2) time: [54.544 µs 55.119 µs 55.720 µs] change: [−15.717% −13.982% −12.279%] (p = 0.00 < 0.05) Performance has improved. Found 2 outliers among 100 measurements (2.00%) 1 (1.00%) low mild 1 (1.00%) high severe Benchmarking first_value evaluate_bench nulls=90%, filter=true, all: Collecting 100 samples in estimated 9.7401 s (10k iteratio first_value evaluate_bench nulls=90%, filter=true, all time: [50.126 µs 50.886 µs 51.703 µs] change: [+0.8379% +4.1713% +7.1024%] (p = 0.00 < 0.05) Change within noise threshold. Found 9 outliers among 100 measurements (9.00%) 6 (6.00%) high mild 3 (3.00%) high severe last_value convert_to_state nulls=0%, filter=false time: [97.957 µs 98.314 µs 98.692 µs] change: [−2.8086% −2.0315% −1.2541%] (p = 0.00 < 0.05) Performance has improved. Found 10 outliers among 100 measurements (10.00%) 7 (7.00%) high mild 3 (3.00%) high severe Benchmarking last_value evaluate_bench nulls=0%, filter=false, first(2): Collecting 100 samples in estimated 7.0582 s (10k iter last_value evaluate_bench nulls=0%, filter=false, first(2) time: [52.692 µs 53.414 µs 54.144 µs] change: [−22.228% −20.636% −19.153%] (p = 0.00 < 0.05) Performance has improved. Found 5 outliers among 100 measurements (5.00%) 1 (1.00%) low mild 2 (2.00%) high mild 2 (2.00%) high severe Benchmarking last_value evaluate_bench nulls=0%, filter=false, all: Collecting 100 samples in estimated 6.9411 s (10k iteration last_value evaluate_bench nulls=0%, filter=false, all time: [49.781 µs 50.226 µs 50.793 µs] change: [−1.9658% −0.1634% +1.5825%] (p = 0.86 > 0.05) No change in performance detected. Found 11 outliers among 100 measurements (11.00%) 1 (1.00%) low mild 6 (6.00%) high mild 4 (4.00%) high severe last_value convert_to_state nulls=0%, filter=true time: [2.0639 µs 2.0781 µs 2.0949 µs] change: [−0.7535% +0.4491% +1.6752%] (p = 0.47 > 0.05) No change in performance detected. Found 11 outliers among 100 measurements (11.00%) 3 (3.00%) low mild 4 (4.00%) high mild 4 (4.00%) high severe Benchmarking last_value evaluate_bench nulls=0%, filter=true, first(2): Collecting 100 samples in estimated 9.8040 s (10k itera last_value evaluate_bench nulls=0%, filter=true, first(2) time: [53.779 µs 54.311 µs 54.868 µs] change: [−15.863% −14.071% −12.391%] (p = 0.00 < 0.05) Performance has improved. Found 7 outliers among 100 measurements (7.00%) 4 (4.00%) high mild 3 (3.00%) high severe Benchmarking last_value evaluate_bench nulls=0%, filter=true, all: Collecting 100 samples in estimated 9.6860 s (10k iterations last_value evaluate_bench nulls=0%, filter=true, all time: [50.276 µs 50.794 µs 51.429 µs] change: [−1.6780% +0.0697% +1.9541%] (p = 0.94 > 0.05) No change in performance detected. Found 6 outliers among 100 measurements (6.00%) 2 (2.00%) high mild 4 (4.00%) high severe last_value convert_to_state nulls=90%, filter=false time: [97.508 µs 98.412 µs 99.486 µs] change: [−1.2994% −0.3404% +0.8319%] (p = 0.52 > 0.05) No change in performance detected. Found 9 outliers among 100 measurements (9.00%) 2 (2.00%) low mild 3 (3.00%) high mild 4 (4.00%) high severe Benchmarking last_value evaluate_bench nulls=90%, filter=false, first(2): Collecting 100 samples in estimated 8.9282 s (10k ite last_value evaluate_bench nulls=90%, filter=false, first(2) time: [54.064 µs 54.748 µs 55.433 µs] change: [−15.790% −13.678% −11.790%] (p = 0.00 < 0.05) Performance has improved. Found 5 outliers among 100 measurements (5.00%) 3 (3.00%) low mild 1 (1.00%) high mild 1 (1.00%) high severe Benchmarking last_value evaluate_bench nulls=90%, filter=false, all: Collecting 100 samples in estimated 9.2411 s (10k iteratio last_value evaluate_bench nulls=90%, filter=false, all time: [49.964 µs 50.731 µs 51.630 µs] change: [−2.4530% −0.2470% +1.8407%] (p = 0.82 > 0.05) No change in performance detected. Found 9 outliers among 100 measurements (9.00%) 8 (8.00%) high mild 1 (1.00%) high severe last_value convert_to_state nulls=90%, filter=true time: [2.0660 µs 2.0874 µs 2.1139 µs] change: [−3.2299% −1.8585% −0.5850%] (p = 0.01 < 0.05) Change within noise threshold. Found 10 outliers among 100 measurements (10.00%) 4 (4.00%) high mild 6 (6.00%) high severe Benchmarking last_value evaluate_bench nulls=90%, filter=true, first(2): Warming up for 3.0000 s Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.6s, enable flat sampling, or reduce sample count to 60. Benchmarking last_value evaluate_bench nulls=90%, filter=true, first(2): Collecting 100 samples in estimated 5.6274 s (5050 ite last_value evaluate_bench nulls=90%, filter=true, first(2) time: [53.565 µs 54.399 µs 55.381 µs] change: [−18.757% −16.680% −14.334%] (p = 0.00 < 0.05) Performance has improved. Found 8 outliers among 100 measurements (8.00%) 8 (8.00%) high mild Benchmarking last_value evaluate_bench nulls=90%, filter=true, all: Warming up for 3.0000 s Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.3s, enable flat sampling, or reduce sample count to 60. Benchmarking last_value evaluate_bench nulls=90%, filter=true, all: Collecting 100 samples in estimated 5.3402 s (5050 iteratio last_value evaluate_bench nulls=90%, filter=true, all time: [49.473 µs 50.055 µs 50.743 µs] change: [−41.448% −32.836% −24.471%] (p = 0.00 < 0.05) Performance has improved. Found 9 outliers among 100 measurements (9.00%) 5 (5.00%) high mild 4 (4.00%) high severe cargo bench --bench first_last -- --baseline main-first_last3 810.16s user 14.72s system 135% cpu 10:09.30 total irix@tau ~/projects/third-party/datafusion [optimise-first_last (theirix/optimise-first_last)?] % time cargo bench --bench first_last -- --baseline main-first_last4 Compiling datafusion-functions-aggregate v53.0.0 (/Users/irix/projects/third-party/datafusion/datafusion/functions-aggregate) Finished `bench` profile [optimized] target(s) in 5m 04s Running benches/first_last.rs (target/release/deps/first_last-c2812404ab64de23) Gnuplot not found, using plotters backend first_value convert_to_state nulls=0%, filter=false time: [100.29 µs 100.92 µs 101.62 µs] change: [−1.3141% −0.4446% +0.4530%] (p = 0.34 > 0.05) No change in performance detected. Found 9 outliers among 100 measurements (9.00%) 7 (7.00%) high mild 2 (2.00%) high severe Benchmarking first_value evaluate_bench nulls=0%, filter=false, first(2): Collecting 100 samples in estimated 7.6058 s (10k ite first_value evaluate_bench nulls=0%, filter=false, first(2) time: [57.325 µs 58.842 µs 60.567 µs] change: [−14.545% −12.765% −10.817%] (p = 0.00 < 0.05) Performance has improved. Found 5 outliers among 100 measurements (5.00%) 3 (3.00%) high mild 2 (2.00%) high severe Benchmarking first_value evaluate_bench nulls=0%, filter=false, all: Collecting 100 samples in estimated 7.1421 s (10k iteratio first_value evaluate_bench nulls=0%, filter=false, all time: [52.632 µs 53.182 µs 53.814 µs] change: [−9.5351% −6.9990% −4.3937%] (p = 0.00 < 0.05) Performance has improved. Found 10 outliers among 100 measurements (10.00%) 1 (1.00%) low mild 6 (6.00%) high mild 3 (3.00%) high severe first_value convert_to_state nulls=0%, filter=true time: [2.2486 µs 2.3289 µs 2.4224 µs] change: [−14.956% −12.528% −9.6570%] (p = 0.00 < 0.05) Performance has improved. Found 19 outliers among 100 measurements (19.00%) 2 (2.00%) high mild 17 (17.00%) high severe Benchmarking first_value evaluate_bench nulls=0%, filter=true, first(2): Warming up for 3.0000 s Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.3s, enable flat sampling, or reduce sample count to 60. Benchmarking first_value evaluate_bench nulls=0%, filter=true, first(2): Collecting 100 samples in estimated 5.3276 s (5050 ite first_value evaluate_bench nulls=0%, filter=true, first(2) time: [60.696 µs 62.295 µs 63.977 µs] change: [−28.986% −27.090% −25.120%] (p = 0.00 < 0.05) Performance has improved. Found 7 outliers among 100 measurements (7.00%) 2 (2.00%) low mild 5 (5.00%) high mild Benchmarking first_value evaluate_bench nulls=0%, filter=true, all: Warming up for 3.0000 s Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.1s, enable flat sampling, or reduce sample count to 60. Benchmarking first_value evaluate_bench nulls=0%, filter=true, all: Collecting 100 samples in estimated 5.1224 s (5050 iteratio first_value evaluate_bench nulls=0%, filter=true, all time: [53.834 µs 54.827 µs 55.913 µs] change: [−12.918% −8.8195% −4.0336%] (p = 0.00 < 0.05) Performance has improved. Found 7 outliers among 100 measurements (7.00%) 3 (3.00%) low mild 1 (1.00%) high mild 3 (3.00%) high severe Benchmarking first_value convert_to_state nulls=90%, filter=false: Collecting 100 samples in estimated 5.1237 s (50k iterations first_value convert_to_state nulls=90%, filter=false time: [104.46 µs 106.26 µs 108.42 µs] change: [−11.589% −8.9599% −6.3106%] (p = 0.00 < 0.05) Performance has improved. Found 6 outliers among 100 measurements (6.00%) 6 (6.00%) high mild Benchmarking first_value evaluate_bench nulls=90%, filter=false, first(2): Collecting 100 samples in estimated 8.1885 s (10k it first_value evaluate_bench nulls=90%, filter=false, first(2) time: [59.528 µs 60.651 µs 61.809 µs] change: [−21.550% −19.288% −16.929%] (p = 0.00 < 0.05) Performance has improved. Found 5 outliers among 100 measurements (5.00%) 5 (5.00%) high mild Benchmarking first_value evaluate_bench nulls=90%, filter=false, all: Collecting 100 samples in estimated 8.2091 s (10k iterati first_value evaluate_bench nulls=90%, filter=false, all time: [53.384 µs 54.251 µs 55.248 µs] change: [−9.1283% −6.6250% −4.4009%] (p = 0.00 < 0.05) Performance has improved. Found 13 outliers among 100 measurements (13.00%) 7 (7.00%) high mild 6 (6.00%) high severe Benchmarking first_value convert_to_state nulls=90%, filter=true: Collecting 100 samples in estimated 5.0021 s (2.4M iterations first_value convert_to_state nulls=90%, filter=true time: [2.1172 µs 2.1355 µs 2.1555 µs] change: [−0.6685% +0.5813% +1.7649%] (p = 0.37 > 0.05) No change in performance detected. Found 7 outliers among 100 measurements (7.00%) 3 (3.00%) high mild 4 (4.00%) high severe Benchmarking first_value evaluate_bench nulls=90%, filter=true, first(2): Warming up for 3.0000 s Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.2s, enable flat sampling, or reduce sample count to 60. Benchmarking first_value evaluate_bench nulls=90%, filter=true, first(2): Collecting 100 samples in estimated 5.2407 s (5050 it first_value evaluate_bench nulls=90%, filter=true, first(2) time: [57.516 µs 58.757 µs 60.089 µs] change: [−18.305% −15.732% −13.092%] (p = 0.00 < 0.05) Performance has improved. Found 5 outliers among 100 measurements (5.00%) 5 (5.00%) high mild Benchmarking first_value evaluate_bench nulls=90%, filter=true, all: Warming up for 3.0000 s Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.3s, enable flat sampling, or reduce sample count to 60. Benchmarking first_value evaluate_bench nulls=90%, filter=true, all: Collecting 100 samples in estimated 5.2574 s (5050 iterati first_value evaluate_bench nulls=90%, filter=true, all time: [63.381 µs 65.433 µs 67.411 µs] change: [+5.8137% +12.240% +19.268%] (p = 0.00 < 0.05) Performance has regressed. Found 10 outliers among 100 measurements (10.00%) 1 (1.00%) high mild 9 (9.00%) high severe last_value convert_to_state nulls=0%, filter=false time: [100.59 µs 101.71 µs 103.06 µs] change: [−0.8042% +0.1307% +1.0555%] (p = 0.79 > 0.05) No change in performance detected. Found 11 outliers among 100 measurements (11.00%) 2 (2.00%) low mild 6 (6.00%) high mild 3 (3.00%) high severe Benchmarking last_value evaluate_bench nulls=0%, filter=false, first(2): Collecting 100 samples in estimated 7.3301 s (10k iter last_value evaluate_bench nulls=0%, filter=false, first(2) time: [59.031 µs 60.067 µs 61.215 µs] change: [−20.562% −15.398% −10.596%] (p = 0.00 < 0.05) Performance has improved. Benchmarking last_value evaluate_bench nulls=0%, filter=false, all: Collecting 100 samples in estimated 7.4307 s (10k iteration last_value evaluate_bench nulls=0%, filter=false, all time: [54.526 µs 55.568 µs 56.708 µs] change: [−9.5126% −5.3244% −1.2387%] (p = 0.02 < 0.05) Performance has improved. Found 3 outliers among 100 measurements (3.00%) 2 (2.00%) high mild 1 (1.00%) high severe last_value convert_to_state nulls=0%, filter=true time: [2.1428 µs 2.1838 µs 2.2289 µs] change: [−5.7061% −3.1097% −0.6074%] (p = 0.02 < 0.05) Change within noise threshold. Benchmarking last_value evaluate_bench nulls=0%, filter=true, first(2): Warming up for 3.0000 s Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.1s, enable flat sampling, or reduce sample count to 70. Benchmarking last_value evaluate_bench nulls=0%, filter=true, first(2): Collecting 100 samples in estimated 5.0694 s (5050 iter last_value evaluate_bench nulls=0%, filter=true, first(2) time: [63.666 µs 66.076 µs 68.373 µs] change: [−17.910% −13.091% −8.0825%] (p = 0.00 < 0.05) Performance has improved. Found 2 outliers among 100 measurements (2.00%) 2 (2.00%) high mild Benchmarking last_value evaluate_bench nulls=0%, filter=true, all: Warming up for 3.0000 s Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.1s, enable flat sampling, or reduce sample count to 60. Benchmarking last_value evaluate_bench nulls=0%, filter=true, all: Collecting 100 samples in estimated 5.1148 s (5050 iteration last_value evaluate_bench nulls=0%, filter=true, all time: [51.817 µs 52.699 µs 53.714 µs] change: [−15.171% −12.649% −10.079%] (p = 0.00 < 0.05) Performance has improved. Found 7 outliers among 100 measurements (7.00%) 1 (1.00%) low mild 3 (3.00%) high mild 3 (3.00%) high severe last_value convert_to_state nulls=90%, filter=false time: [107.51 µs 110.89 µs 114.78 µs] change: [−0.7613% +1.8668% +4.7036%] (p = 0.17 > 0.05) No change in performance detected. Found 3 outliers among 100 measurements (3.00%) 3 (3.00%) high mild Benchmarking last_value evaluate_bench nulls=90%, filter=false, first(2): Collecting 100 samples in estimated 9.6846 s (10k ite last_value evaluate_bench nulls=90%, filter=false, first(2) time: [58.180 µs 58.969 µs 59.915 µs] change: [−19.674% −16.125% −12.269%] (p = 0.00 < 0.05) Performance has improved. Found 2 outliers among 100 measurements (2.00%) 1 (1.00%) high mild 1 (1.00%) high severe Benchmarking last_value evaluate_bench nulls=90%, filter=false, all: Collecting 100 samples in estimated 9.2340 s (10k iteratio last_value evaluate_bench nulls=90%, filter=false, all time: [52.432 µs 53.242 µs 54.103 µs] change: [−12.306% −9.3583% −6.3687%] (p = 0.00 < 0.05) Performance has improved. Found 8 outliers among 100 measurements (8.00%) 7 (7.00%) high mild 1 (1.00%) high severe last_value convert_to_state nulls=90%, filter=true time: [2.0731 µs 2.0901 µs 2.1137 µs] change: [−10.926% −9.9495% −8.9933%] (p = 0.00 < 0.05) Performance has improved. Found 8 outliers among 100 measurements (8.00%) 3 (3.00%) low mild 2 (2.00%) high mild 3 (3.00%) high severe Benchmarking last_value evaluate_bench nulls=90%, filter=true, first(2): Warming up for 3.0000 s Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.7s, enable flat sampling, or reduce sample count to 60. Benchmarking last_value evaluate_bench nulls=90%, filter=true, first(2): Collecting 100 samples in estimated 5.7484 s (5050 ite last_value evaluate_bench nulls=90%, filter=true, first(2) time: [56.238 µs 57.148 µs 58.202 µs] change: [−31.872% −29.982% −27.838%] (p = 0.00 < 0.05) Performance has improved. Found 7 outliers among 100 measurements (7.00%) 4 (4.00%) high mild 3 (3.00%) high severe Benchmarking last_value evaluate_bench nulls=90%, filter=true, all: Warming up for 3.0000 s Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.8s, enable flat sampling, or reduce sample count to 60. Benchmarking last_value evaluate_bench nulls=90%, filter=true, all: Collecting 100 samples in estimated 5.7562 s (5050 iteratio last_value evaluate_bench nulls=90%, filter=true, all time: [51.057 µs 52.223 µs 53.542 µs] change: [−21.819% −19.372% −16.998%] (p = 0.00 < 0.05) Performance has improved. Found 7 outliers among 100 measurements (7.00%) 1 (1.00%) low mild 4 (4.00%) high mild 2 (2.00%) high severe cargo bench --bench first_last -- --baseline main-first_last4 786.76s user 16.16s system 135% cpu 9:50.80 total </details> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
…onger maintained (apache#21030) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes apache#20863 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? - Replace abi_stable types with stabby equivalents: - `RVec` -> `stabby::vec::Vec` - `RString` -> `stabby::string::String` - Add custom `FFI_Option` and `FFI_Result` types since we have raw pointers and self referential pointers which can't implement `IStable` trait needed for stabby - Rename `macros: rresult` → `sresult`, `rresult_return `→ `sresult_return` - Fix library path detection for debug/release builds in the example loader - Update README <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? Yes . With df python bindings on DF 52 release <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->
…cans (apache#20844) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> I figured it will be easier to submit PR right away as change doesn't look controversial. I'm happy to create an issue and link it here if you'd prefer. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> So short story is: in another project we'd like to use DataFusion's to "build" operations on data and then submit resulting logical plan _somewhere_ to execute (likely not using DF to actually execute the query). Since those plans never meant to be executed by DF we use `EmptyTable` as a base to bring schema to DF without any actual data. `EmptyTable` scans not being serializable prevents us from sending those plans to Python or over the wire. I believe this change makes datafusion's LogicalPlan more portable and more usable outside of datafusion's query executor. Longer story: [VegaFusion](https://github.com/vega/vegafusion) does server-side aggregation for Vega charts and is powered by DataFusion. We recently added option to [use custom query/plan executors](vega/vegafusion#573), which allows user to pass a schema (without data) to VegaFusion which will add all necessary aggregations (but not execute them) and return a logical plan to user. They can then outsource this plan to custom query executor (e.g. Spark). This is already implemented and works. However, since VegaFusion is most commonly used through Python bindings, we'd like to expose this API to Python too (and additionally as part of gPRC API too) , which requires serializing built plans to protobuf. Currently we use `EmptyTable` to bring schema without any data to DataFusion. But since it can't be converted to protobuf, we're unable to expose this API. We considered providing custom decoder/encoder, but that would work only for gRPC case, but not Python as datafusion-python doesn't allow to provide custom decoder as far as I understand. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> * Moved `EmptyTable` from `datafusion-core` into `datafusion-catalog` and added backwards compatibility re-export (following pattern for other table providers moved earlier) * Added new `EmptyTableScanNode` to protobuf definitions * Added encoding and decoding for new entity into `AsLogicalPlan for LogicalPlanNode` implementation ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> I added two roundtrip tests for the new node ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> `EmptyTable` can be imported from `datafusion-catalog` crate now, but old crate (`datafusion-core`) still re-exports it, so this shouldn't be breaking change <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> P.S. Just to be explicit, code itself was written mostly by LLM (as I'm not that proficient in Rust yet). I did review and test it though
## Which issue does this PR close? * Closes apache#21150. --- ## Rationale for this change The existing implementation of `min`/`max` does not correctly support dictionary-encoded arrays. Previously, dictionary arrays were handled by directly evaluating their underlying values array, which is semantically incorrect because: * It may include unreferenced values that do not appear in the logical dataset * It ignores nulls in the key array * It does not preserve dictionary key semantics in scalar results This leads to incorrect aggregation results for dictionary types. This PR introduces a logical row-based evaluation for dictionary arrays and ensures scalar comparisons correctly unwrap and rewrap dictionary values when needed. --- ## What changes are included in this PR? * Add logical row-based min/max computation (`scalar_row_extreme`) for: * Dictionary arrays * Struct, List, LargeList, and FixedSizeList types * Replace previous dictionary handling that operated on `values()` with correct row-wise evaluation * Introduce `requires_logical_row_scan` to centralize fallback logic for complex types * Enhance scalar comparison logic: * Unwrap dictionary scalars before comparison * Rewrap results when both inputs are dictionaries with matching key types * Validate key type compatibility * Improve error messaging for incompatible scalar comparisons * Remove obsolete `min_max_batch_generic` implementation --- ## Are these changes tested? Yes. Comprehensive tests have been added to validate correctness across multiple scenarios: * Basic dictionary min/max behavior * Handling of null keys and null values * Ignoring unreferenced dictionary values * Multi-batch aggregation behavior * Float dictionary handling including `NaN` and infinities These tests ensure correctness and guard against regressions. --- ## Are there any user-facing changes? Yes. The behavior of `min` and `max` on dictionary-encoded arrays is now: * Correct and semantically aligned with logical row values * Consistent with other data types Previously incorrect results may now differ, which is a correctness fix rather than a breaking API change. --- ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed and tested.
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> N/A ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> <img width="302" height="45" alt="image" src="https://github.com/user-attachments/assets/fed0ded5-5ef5-4e10-9bd6-bb3391470ee7" /> Lets work on getting this number down to 0 ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> Fix some permissions to be minimum required in GH action workflows ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> All affected workflows were run on this commit: apache@48ffe3f - Since some workflows trigger on certain changes only - large files successful: https://github.com/apache/datafusion/actions/runs/24923629680/job/72989644694?pr=21838 - docs pr successful: https://github.com/apache/datafusion/actions/runs/24923629679/job/72989644704?pr=21838 - dependencies: - circular check successful: https://github.com/apache/datafusion/actions/runs/24923629662/job/72989644657?pr=21838 - unused check successful: https://github.com/apache/datafusion/actions/runs/24923629662/job/72989644656?pr=21838 - audit successful: https://github.com/apache/datafusion/actions/runs/24923629664/job/72989644667?pr=21838 - dev: - license check successful: https://github.com/apache/datafusion/actions/runs/24923629678/job/72989644714?pr=21838 - prettier successful: https://github.com/apache/datafusion/actions/runs/24923629678/job/72989644708?pr=21838 - asf yaml check successful: https://github.com/apache/datafusion/actions/runs/24923629678/job/72989644706?pr=21838 - typos successful: https://github.com/apache/datafusion/actions/runs/24923629678/job/72989644701?pr=21838 ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
) Refs apache#15986. **Why:** `spark_hex` walked one nibble at a time — two `HEX_CHARS[i]` lookups and two `Vec::push` calls per input byte. The hot loop flattens into one indexed load and one `extend_from_slice` per byte with a precomputed table. **What changed:** added `HEX_LOOKUP_LOWER` / `HEX_LOOKUP_UPPER` as `[[u8; 2]; 256]` const tables built at compile time. Bytes path now does a single lookup + 2-byte extend per input byte. The int64 path consumes two nibbles per iteration via the same table, with a fall-through for the high nibble. Behaviour for `0`, `i64::MAX`, `i64::MIN`, `-1` preserved. **Tests:** extended `test_hex_int64` to cover edge values; new `test_hex_lookup_table_covers_all_bytes` cross-checks every entry against `format!("{:02X/x}")`; new `test_spark_hex_binary_round_trip_all_bytes` feeds all 256 byte values through `spark_hex` and verifies the result. `cargo test -p datafusion-spark --lib hex` → 8 pass. `cargo clippy --all-features --all-targets` clean. `cargo bench --no-run` builds — existing `benches/hex.rs` already covers Int64/Utf8/Utf8View/LargeUtf8/Binary/LargeBinary plus dict paths. **Not in this PR:** the apache#15947 review also flagged Utf8View output and dictionary-key reuse — those felt worth their own PRs to keep this focused on the per-byte hot path. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
This a clean version of apache#18921 to make it easier to review **this is a breaking change due to adding variant to `Expr` enum, new methods on traits `Session`, `FunctionRegistry` and `ContextProvider` and a new arg on `TaskContext::new`** This PR adds support for lambdas and the `array_transform` function used to test the lambda implementation. Example usage: ```sql SELECT array_transform([2, 3], v -> v != 2); [false, true] -- arbitrally nested lambdas are also supported SELECT array_transform([[[2, 3]]], m -> array_transform(m, l -> array_transform(l, v -> v*2))); [[[4, 6]]] ``` Note: column capture has been removed for now and will be added on a follow on PR, see apache#21172 Some comments on code snippets of this doc show what value each struct, variant or field would hold after planning the first example above. Some literals are simplified pseudo code 3 new `Expr` variants are added, `HigherOrderFunction`, owing a new trait `HigherOrderUDF`, which is like a `ScalarFunction`/`ScalarUDFImpl` with support for lambdas, `Lambda`, for the lambda body and it's parameters names, and `LambdaVariable`, which is like `Column` but for lambdas parameters. Their logical representations: ```rust enum Expr { // array_transform([2, 3], v -> v != 2) HigherOrderFunction(HigherOrderFunction), // v -> v != 2 Lambda(Lambda), // v, of the lambda body: v != 2 LambdaVariable(LambdaVariable), ... } // array_transform([2, 3], v -> v != 2) struct HigherOrderFunction { // global instance of array_transform pub func: Arc<dyn HigherOrderUDF>, // [Expr::ScalarValue([2, 3]), Expr::Lambda(v -> v != 2)] pub args: Vec<Expr>, } // v -> v != 2 struct Lambda { // ["v"] pub params: Vec<String>, // v != 2 pub body: Box<Expr>, } // v, of the lambda body: v != 2 struct LambdaVariable { // "v" pub name: String, // Field::new("", DataType::Int32, false) // Note: a follow on PR will make this field optional // to free expr_api from specifying it beforehand, // and add resolve_lambda_variables method to Expr, // similar to Expr::Placeholder, see apache#21172 pub field: FieldRef, pub spans: Spans, } ``` The example would be planned into a tree like this: ``` HigherOrderFunctionExpression name: array_transform children: 1. ListExpression [2,3] 2. LambdaExpression parameters: ["v"] body: BinaryExpression (!=) left: LambdaVariableExpression("v", Field::new("", Int32, false)) right: LiteralExpression("2") ``` The physical counterparts definition: ```rust struct HigherOrderFunctionExpr { // global instance of array_transform fun: Arc<dyn HigherOrderUDF>, // "array_transform" name: String, // [LiteralExpr([2, 3], LambdaExpr("v -> v != 2"))] args: Vec<Arc<dyn PhysicalExpr>>, // [1], the positions at args that contains lambdas lambda_positions: Vec<usize>, // Field::new("", DataType::new_list(DataType::Boolean, false), false) return_field: FieldRef, config_options: Arc<ConfigOptions>, } struct LambdaExpr { // ["v"] params: Vec<String>, // v -> v != 2 body: Arc<dyn PhysicalExpr>, } struct LambdaVariable { // Field::new("v", DataType::Int32, false) field: FieldRef, // 0, the first and only parameter, "v" index: usize, } ``` Note: For those who primarly wants to check if this lambda implementation supports their usecase and don't want to spend much time here, it's okay to skip most collapsed blocks, as those serve mostly to help code reviewers, with the exception of `HigherOrderUDF` and the `array_transform` implementation of `HigherOrderUDF` relevant methods, collapsed due to their size The added `HigherOrderUDF` trait is almost a clone of `ScalarUDFImpl`, with the exception of: 1. `return_field_from_args` and `invoke_with_args`, where now `args.args` is a list of enums with two variants: `Value` or `Lambda` instead of a list of values 2. the addition of `lambda_parameters`, which return a `Field` for each parameter supported for every lambda argument based on the `Field` of the non lambda arguments 3. the removal of `return_field` and the deprecated ones `is_nullable` and `display_name`. 4. Not yet includes analogues to the methods preimage, placement, evaluate_bounds, propagate_constraints, output_ordering and preserves_lex_ordering <details><summary>HigherOrderUDF</summary> ```rust trait HigherOrderUDF { /// Return the field of all the parameters supported by all the supported lambdas of this function /// based on the field of the value arguments. If a lambda support multiple parameters, or if multiple /// lambdas are supported and some are optional, all should be returned, /// regardless of whether they are used on a particular invocation /// /// Tip: If you have a [`HigherOrderFunction`] invocation, you can call the helper /// [`HigherOrderFunction::lambda_parameters`] instead of this method directly /// /// [`HigherOrderFunction`]: crate::expr::HigherOrderFunction /// [`HigherOrderFunction::lambda_parameters`]: crate::expr::HigherOrderFunction::lambda_parameters /// /// Example for array_transform: /// /// `array_transform([2.0, 8.0], v -> v > 4.0)` /// /// ```ignore /// let lambda_parameters = array_transform.lambda_parameters(&[ /// Arc::new(Field::new("", DataType::new_list(DataType::Float32, false))), // the Field of the literal `[2, 8]` /// ])?; /// /// assert_eq!( /// lambda_parameters, /// vec![ /// // the lambda supported parameters, regardless of how many are actually used /// vec![ /// // the value being transformed /// Field::new("", DataType::Float32, false), /// ] /// ] /// ) /// ``` /// /// The implementation can assume that some other part of the code has coerced /// the actual argument types to match [`Self::signature`]. fn lambda_parameters(&self, value_fields: &[FieldRef]) -> Result<Vec<Vec<Field>>>; fn return_field_from_args(&self, args: LambdaReturnFieldArgs) -> Result<FieldRef>; fn invoke_with_args(&self, args: HigherOrderFunctionArgs) -> Result<ColumnarValue>; // ... omitted methods that are similar in ScalarUDFImpl } /// An argument to a lambda function #[derive(Clone, Debug, PartialEq, Eq)] pub enum ValueOrLambda<V, L> { /// A value with associated data Value(V), /// A lambda with associated data Lambda(L), } /// Information about arguments passed to the function /// /// This structure contains metadata about how the function was called /// such as the type of the arguments, any scalar arguments and if the /// arguments can (ever) be null /// /// See [`HigherOrderUDF::return_field_from_args`] for more information #[derive(Clone, Debug)] pub struct LambdaReturnFieldArgs<'a> { /// The data types of the arguments to the function /// /// If argument `i` to the function is a lambda, it will be the field of the result of the /// lambda if evaluated with the parameters returned from [`HigherOrderUDF::lambda_parameters`] /// /// For example, with `array_transform([1], v -> v == 5)` /// this field will be `[ /// ValueOrLambda::Value(Field::new("", DataType::List(DataType::Int32), false)), /// ValueOrLambda::Lambda(Field::new("", DataType::Boolean, false)) /// ]` pub arg_fields: &'a [ValueOrLambda<FieldRef, FieldRef>], /// Is argument `i` to the function a scalar (constant)? /// /// If the argument `i` is not a scalar, it will be None /// /// For example, if a function is called like `array_transform([1], v -> v == 5)` /// this field will be `[Some(ScalarValue::List(...), None]` pub scalar_arguments: &'a [Option<&'a ScalarValue>], } /// Arguments passed to [`HigherOrderUDF::invoke_with_args`] when invoking a /// lambda function. #[derive(Debug, Clone)] pub struct HigherOrderFunctionArgs { /// The evaluated arguments and lambdas to the function pub args: Vec<ValueOrLambda<ColumnarValue, LambdaArgument>>, /// Field associated with each arg, if it exists /// For lambdas, it will be the field of the result of /// the lambda if evaluated with the parameters /// returned from [`HigherOrderUDF::lambda_parameters`] pub arg_fields: Vec<ValueOrLambda<FieldRef, FieldRef>>, /// The number of rows in record batch being evaluated pub number_rows: usize, /// The return field of the lambda function returned /// (from `return_field_from_args`) when creating the /// physical expression from the logical expression pub return_field: FieldRef, /// The config options at execution time pub config_options: Arc<ConfigOptions>, } /// A lambda argument to a HigherOrderFunction #[derive(Clone, Debug)] pub struct LambdaArgument { /// The parameters defined in this lambda /// /// For example, for `array_transform([2], v -> -v)`, /// this will be `vec![Field::new("v", DataType::Int32, true)]` params: Vec<FieldRef>, /// The body of the lambda /// /// For example, for `array_transform([2], v -> -v)`, /// this will be the physical expression of `-v` body: Arc<dyn PhysicalExpr>, } impl LambdaArgument { /// Evaluate this lambda /// `args` should evalute to the value of each parameter /// of the correspondent lambda returned in [HigherOrderUDF::lambda_parameters]. pub fn evaluate( &self, args: &[&dyn Fn() -> Result<ArrayRef>], ) -> Result<ColumnarValue> { let columns = args .iter() .take(self.params.len()) .map(|arg| arg()) .collect::<Result<_>>()?; let schema = Arc::new(Schema::new(self.params.clone())); let batch = RecordBatch::try_new(schema, columns)?; self.body.evaluate(&batch) } } ``` </details> <details><summary>array_transform lambda_parameters implementation</summary> ```rust impl HigherOrderUDF for ArrayTransform { fn lambda_parameters(&self, value_fields: &[FieldRef]) -> Result<Vec<Vec<Field>>> { let list = if value_fields.len() == 1 { &value_fields[0] } else { return plan_err!( "{} function requires 1 value arguments, got {}", self.name(), value_fields.len() ); }; let field = match list.data_type() { DataType::List(field) => field, DataType::LargeList(field) => field, DataType::FixedSizeList(field, _) => field, _ => return plan_err!("expected list, got {list}"), }; // we don't need to check whether the lambda contains more than two parameters, // e.g. array_transform([], (v, i, j) -> v+i+j), as datafusion will do that for us let value = Field::new("", field.data_type().clone(), field.is_nullable()) .with_metadata(field.metadata().clone()); Ok(vec![vec![value]]) } } ``` </details> <details><summary>array_transform return_field_from_args implementation</summary> ```rust fn value_lambda_pair<'a, V: Debug, L: Debug>( name: &str, args: &'a [ValueOrLambda<V, L>], ) -> Result<(&'a V, &'a L)> { let [value, lambda] = take_function_args(name, args)?; let (ValueOrLambda::Value(value), ValueOrLambda::Lambda(lambda)) = (value, lambda) else { return plan_err!( "{name} expects a value followed by a lambda, got {value:?} and {lambda:?}" ); }; Ok((value, lambda)) } impl HigherOrderUDF for ArrayTransform { fn return_field_from_args( &self, args: HigherOrderReturnFieldArgs, ) -> Result<Arc<Field>> { let (list, lambda) = value_lambda_pair(self.name(), args.arg_fields)?; // lambda is the resulting field of executing the lambda body // with the parameters returned in lambda_parameters let field = Arc::new(Field::new( Field::LIST_FIELD_DEFAULT_NAME, lambda.data_type().clone(), lambda.is_nullable(), )); let return_type = match list.data_type() { DataType::List(_) => DataType::List(field), DataType::LargeList(_) => DataType::LargeList(field), DataType::FixedSizeList(_, size) => DataType::FixedSizeList(field, *size), other => plan_err!("expected list, got {other}")?, }; Ok(Arc::new(Field::new("", return_type, list.is_nullable()))) } } ``` </details> <details><summary>array_transform invoke_with_args implementation</summary> ```rust impl HigherOrderUDF for ArrayTransform { fn invoke_with_args(&self, args: HigherOrderFunctionArgs) -> Result<ColumnarValue> { let (list, lambda) = value_lambda_pair(self.name(), &args.args)?; let list_array = list.to_array(args.number_rows)?; // Fast path for fully null input array and also the only way to safely work with // a fully null fixed size list array as it can't be handled by remove_list_null_values below if list_array.null_count() == list_array.len() { return Ok(ColumnarValue::Array(new_null_array( args.return_type(), list_array.len(), ))); } // as per list_values docs, if list_array is sliced, list_values will be sliced too, // so before constructing the transformed array below, we must adjust the list offsets with // adjust_offsets_for_slice let list_values = list_values(&list_array)?; // by passing closures, lambda.evaluate can evaluate only those actually needed let values_param = || Ok(Arc::clone(&list_values)); // call the transforming lambda let transformed_values = lambda .evaluate(&[&values_param])? .into_array(list_values.len())?; let field = match args.return_field.data_type() { DataType::List(field) | DataType::LargeList(field) | DataType::FixedSizeList(field, _) => Arc::clone(field), _ => { return exec_err!( "{} expected ScalarFunctionArgs.return_field to be a list, got {}", self.name(), args.return_field ); } }; let transformed_list = match list_array.data_type() { DataType::List(_) => { let list = list_array.as_list(); // since we called list_values above which would return sliced values for // a sliced list, we must adjust the offsets here as otherwise they would be invalid let adjusted_offsets = adjust_offsets_for_slice(list); Arc::new(ListArray::new( field, adjusted_offsets, transformed_values, list.nulls().cloned(), )) as ArrayRef } DataType::LargeList(_) => { let large_list = list_array.as_list(); // since we called list_values above which would return sliced values for // a sliced list, we must adjust the offsets here as otherwise they would be invalid let adjusted_offsets = adjust_offsets_for_slice(large_list); Arc::new(LargeListArray::new( field, adjusted_offsets, transformed_values, large_list.nulls().cloned(), )) } DataType::FixedSizeList(_, value_length) => { Arc::new(FixedSizeListArray::new( field, *value_length, transformed_values, list_array.as_fixed_size_list().nulls().cloned(), )) } other => exec_err!("expected list, got {other}")?, }; Ok(ColumnarValue::Array(transformed_list)) } } ``` </details> <details><summary>How relevant HigherOrderUDF methods would be called and what they would return during planning and evaluation of the example</summary> ```rust // this is called at sql planning let lambda_parameters = lambda_udf.lambda_parameters(&[ Field::new("", DataType::new_list(DataType::Int32, false), false), // the Field of the [2, 3] literal ])?; assert_eq!( lambda_parameters, vec![ // the parameters that *can* be declared on the lambda, and not only // those actually declared: the implementation doesn't need to care // about it vec![ Field::new("", DataType::Int32, false), // the list inner value ]] ); // this is called every time ExprSchemable is called on a HigherOrderFunction let return_field = array_transform.return_field_from_args(&LambdaReturnFieldArgs { arg_fields: &[ ValueOrLambda::Value(Field::new("", DataType::new_list(DataType::Int32, false), false)), ValueOrLambda::Lambda(Field::new("", DataType::Boolean, false)), // the return_field of the expression "v != 2" when "v" is of the type returned in lambda_parameters ], scalar_arguments // irrelevant })?; assert_eq!(return_field, Field::new("", DataType::new_list(DataType::Boolean, false), false)); let value = array_transform.evaluate(&HigherOrderFunctionArgs { args: vec![ ValueOrLambda::Value(List([2, 3])), ValueOrLambda::Lambda(LambdaArgument of `v -> v != 2`), ], arg_fields, // same as above number_rows: 1, return_field, // same as above config_options, // irrelevant })?; assert_eq!(value, BooleanArray::from([false, true])) ``` </details> <br> <br> A pair HigherOrderUDF/HigherOrderUDFImpl like ScalarFunction was not used because those exist only [to maintain backwards compatibility with the older API](https://docs.rs/datafusion/latest/datafusion/logical_expr/struct.ScalarUDF.html#api-note) apache#8045 </details> <br> Why `LambdaVariable` and not `Column`: Existing tree traversals that operate on columns would break if some column nodes referenced to a lambda parameter and not a real column. In the example query, projection pushdown would try to push the lambda parameter "v", which won't exist in table "t". Example of code of another traversal that would break: ```rust fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter { let mut used_columns = HashSet::new(); expr.apply(|expr| { if let Some(col) = expr.as_any().downcast_ref::<Column>() { // if this is a lambda column, this function will break used_columns.insert(col.index()); } Ok(TreeNodeRecursion::Continue) }); ... } ``` Furthermore, the implemention of `ExprSchemable` and `PhysicalExpr::return_field` for `Column` expects that the schema it receives as a argument contains an entry for its name, which is not the case for lambda parameters. By including a `FieldRef` on `LambdaVariable` that should be resolved during construction time in the sql planner, `ExprSchemable` and `PhysicalExpr::return_field` simply return it's own Field: <details><summary>LambdaVariable ExprSchemable and PhysicalExpr::return_field implementation </summary> ```rust impl ExprSchemable for Expr { fn to_field( &self, schema: &dyn ExprSchema, ) -> Result<(Option<TableReference>, Arc<Field>)> { let (relation, schema_name) = self.qualified_name(); let field = match self { Expr::LambdaVariable(l) => Ok(Arc::clone(&l.field)), ... }?; Ok(( relation, Arc::new(field.as_ref().clone().with_name(schema_name)), )) } ... } impl PhysicalExpr for LambdaVariable { fn return_field(&self, _input_schema: &Schema) -> Result<FieldRef> { Ok(Arc::clone(&self.field)) } ... } ``` </details> <br> <details><summary>Possible alternatives discarded due to complexity, requiring downstream changes and implementation size:</summary> 1. Add a new set of TreeNode methods that provides the set of lambdas parameters names seen during the traversal, so column nodes can be tested if they refer to a regular column or to a lambda parameter. Any downstream user that wants to support lambdas would need use those methods instead of the existing ones. This also would add 1k+ lines to the PR. ```rust impl Expr { pub fn transform_with_lambdas_params< F: FnMut(Self, &HashSet<String>) -> Result<Transformed<Self>>, >( self, mut f: F, ) -> Result<Transformed<Self>> {} } ``` How minimize_join_filter would looks like: ```rust fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter { let mut used_columns = HashSet::new(); expr.apply_with_lambdas_params(|expr, lambdas_params| { if let Some(col) = expr.as_any().downcast_ref::<Column>() { // dont include lambdas parameters if !lambdas_params.contains(col.name()) { used_columns.insert(col.index()); } } Ok(TreeNodeRecursion::Continue) }) ... } ``` 2. Add a flag to the Column node indicating if it refers to a lambda parameter. Still requires checking for it on existing tree traversals that works on Columns (30+) and also downstream. ```rust //logical struct Column { pub relation: Option<TableReference>, pub name: String, pub spans: Spans, pub is_lambda_parameter: bool, } //physical struct Column { name: String, index: usize, is_lambda_parameter: bool, } ``` How minimize_join_filter would look like: ```rust fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter { let mut used_columns = HashSet::new(); expr.apply(|expr| { if let Some(col) = expr.as_any().downcast_ref::<Column>() { // dont include lambdas parameters if !col.is_lambda_parameter { used_columns.insert(col.index()); } } Ok(TreeNodeRecursion::Continue) }) ... } ``` 1. Add a new set of TreeNode methods that provides a schema that includes the lambdas parameters for the scope of the node being visited/transformed: ```rust impl Expr { pub fn transform_with_schema< F: FnMut(Self, &DFSchema) -> Result<Transformed<Self>>, >( self, schema: &DFSchema, f: F, ) -> Result<Transformed<Self>> { ... } ... other methods } ``` For any given HigherOrderFunction found during the traversal, a new schema is created for each lambda argument that contains it's parameter, returned from HigherOrderUDF::lambda_parameters How it would look like: ```rust pub fn infer_placeholder_types(self, schema: &DFSchema) -> Result<(Expr, bool)> { let mut has_placeholder = false; // Provide the schema as the first argument. // Transforming closure receive an adjusted_schema as argument self.transform_with_schema(schema, |mut expr, adjusted_schema| { match &mut expr { // Default to assuming the arguments are the same type Expr::BinaryExpr(BinaryExpr { left, op: _, right }) => { // use adjusted_schema and not schema. Those expressions may contain // columns referring to a lambda parameter, which Field would only be // available in adjusted_schema and not in schema rewrite_placeholder(left.as_mut(), right.as_ref(), adjusted_schema)?; rewrite_placeholder(right.as_mut(), left.as_ref(), adjusted_schema)?; } .... ``` 2. Make available trought LogicalPlan and ExecutionPlan nodes a schema that includes all lambdas parameters from all expressions owned by the node, and use this schema for tree traversals. For nodes which won't own any expression, the regular schema can be returned ```rust impl LogicalPlan { fn lambda_extended_schema(&self) -> &DFSchema; } trait ExecutionPlan { fn lambda_extended_schema(&self) -> &DFSchema; } //usage impl LogicalPlan { pub fn replace_params_with_values( self, param_values: &ParamValues, ) -> Result<LogicalPlan> { self.transform_up_with_subqueries(|plan| { // use plan.lambda_extended_schema() containing lambdas parameters // instead of plan.schema() which wont let lambda_extended_schema = Arc::clone(plan.lambda_extended_schema()); let name_preserver = NamePreserver::new(&plan); plan.map_expressions(|e| { // if this expression is child of lambda and contain columns referring it's parameters // the lambda_extended_schema already contain them let (e, has_placeholder) = e.infer_placeholder_types(&lambda_extended_schema)?; .... ``` </details> <br> --------- Co-authored-by: Raz Luvaton <16746759+rluvaton@users.noreply.github.com> Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com> Co-authored-by: Lía Adriana <lia.castaneda@datadoghq.com>
…HERE` instead of moving to `JOIN ON` (apache#21694) ## Which issue does this PR close? Partially addresses apache#13156 (inner joins only; outer joins require additional work) When the DataFusion optimizer pushes filter predicates into `TableScan` nodes (e.g. via `FilterPushdown`), the unparser's `try_transform_to_simple_table_scan_with_filters` extracts those filters and then always folds them into the `JOIN ON` clause. This is problematic when the extracted filters contain subquery expressions (scalar subqueries, `IN`, `EXISTS`), because some SQL backends — notably BigQuery — reject subqueries inside `JOIN ON` predicates. This currently breaks 5 TPC-H queries (Q2, Q16, Q17, Q18, Q21) when unparsed SQL is sent to BigQuery. We did attempt to fix this (apache#13496) by moving **all** filters to `WHERE`, which broke `LEFT`/`RIGHT`/`FULL` join semantics (moving a filter from `ON` to `WHERE` changes the result for outer joins, as demonstrated in apache#13132). ## What changes are included in this PR? For **inner joins only**, `table_scan_filters` extracted by `try_transform_to_simple_table_scan_with_filters` are now placed in the `WHERE` clause instead of the `JOIN ON` clause. This is safe because `ON` and `WHERE` are semantically equivalent for inner joins. For non-inner joins (`LEFT`, `RIGHT`, `FULL`), the existing behavior is preserved — filters remain in `JOIN ON` — since moving them to `WHERE` would change query semantics. ## Are these changes tested? Yes. - Added a test case in `test_join_with_table_scan_filters` that constructs an inner join where the right side has a `table_scan_with_filters` containing a scalar subquery. Verifies the subquery predicate appears in `WHERE`, not `JOIN ON`. - Updated existing snapshots in `test_join_with_table_scan_filters` reflecting that `table_scan_filters` now appear in `WHERE` for inner joins. ## Are there any user-facing changes? SQL generated by the unparser for inner joins may now place `TableScan` pushdown filters in the `WHERE` clause instead of the `JOIN ON` clause (similar to changes in `test_join_with_table_scan_filters`) ## Alternatives considered An alternative approach considered is to introduce a `supports_subquery_in_join_predicate` dialect flag that only moves subquery-containing filters to `WHERE` when the dialect opts in (e.g. `BigQueryDialect`), preserving existing behavior for all other dialects. Example implementation: spiceai#151 Current approach was chosen due to - simplicity: no new dialect flag, fewer code paths, simple change. - performance (potentially): as noted in [apache#13156](apache#13156), placing filters in `WHERE` can trigger filter pushdown on the target backend, which is a potential performance win.
## Which issue does this PR close? - Closes apache#21763 ## Rationale for this change Add support for nested types to the `nullif` UDF. ## Are these changes tested? Unit tests included. ## Are there any user-facing changes? No changes to the function's signature. --------- Co-authored-by: Gabriel <45515538+gabotechs@users.noreply.github.com>
…#21877) ## Which issue does this PR close? - Closes apache#21876. ## Rationale for this change As with other recent optimizations, we can optimize NULL handling in `substr_index` by using the new bulk-NULL string builders. Benchmarks: Utf8 - utf8_100_array_long_delimiter: 10.0 µs → 10.1 µs (+1.00%) - utf8_100_array_single_delimiter: 2.9 µs → 2.5 µs (−13.79%) - utf8_100_scalar_long_delimiter_neg: 4.1 µs → 3.5 µs (−14.63%) - utf8_100_scalar_long_delimiter_pos: 2.9 µs → 2.7 µs (−6.90%) - utf8_100_scalar_single_delimiter_neg: 2.2 µs → 1.993 µs (−9.41%) - utf8_100_scalar_single_delimiter_pos: 2.1 µs → 1.845 µs (−12.13%) - utf8_1000_array_long_delimiter: 101.0 µs → 101.1 µs (+0.10%) - utf8_1000_array_single_delimiter: 36.8 µs → 31.7 µs (−13.86%) - utf8_1000_scalar_long_delimiter_neg: 38.9 µs → 36.9 µs (−5.14%) - utf8_1000_scalar_long_delimiter_pos: 25.1 µs → 23.3 µs (−7.17%) - utf8_1000_scalar_single_delimiter_neg: 19.3 µs → 17.7 µs (−8.29%) - utf8_1000_scalar_single_delimiter_pos: 18.2 µs → 16.6 µs (−8.79%) - utf8_10000_array_long_delimiter: 1083.4 µs → 1038.2 µs (−4.17%) - utf8_10000_array_single_delimiter: 461.8 µs → 414.7 µs (−10.20%) - utf8_10000_scalar_long_delimiter_neg: 392.4 µs → 379.3 µs (−3.34%) - utf8_10000_scalar_long_delimiter_pos: 246.5 µs → 227.4 µs (−7.75%) - utf8_10000_scalar_single_delimiter_neg: 191.3 µs → 177.5 µs (−7.21%) - utf8_10000_scalar_single_delimiter_pos: 179.4 µs → 168.8 µs (−5.91%) Utf8View - utf8view_100_array_long_delimiter: 9.5 µs → 9.8 µs (+3.16%) - utf8view_100_array_single_delimiter: 2.6 µs → 2.6 µs (0.00%) - utf8view_100_scalar_long_delimiter_neg: 4.0 µs → 4.0 µs (0.00%) - utf8view_100_scalar_long_delimiter_pos: 2.8 µs → 2.8 µs (0.00%) - utf8view_100_scalar_single_delimiter_neg: 2.3 µs → 2.3 µs (0.00%) - utf8view_100_scalar_single_delimiter_pos: 2.2 µs → 2.1 µs (−4.55%) - utf8view_1000_array_long_delimiter: 94.8 µs → 99.2 µs (+4.64%) - utf8view_1000_array_single_delimiter: 31.5 µs → 32.0 µs (+1.59%) - utf8view_1000_scalar_long_delimiter_neg: 38.7 µs → 39.0 µs (+0.78%) - utf8view_1000_scalar_long_delimiter_pos: 25.4 µs → 25.4 µs (0.00%) - utf8view_1000_scalar_single_delimiter_neg: 21.4 µs → 21.8 µs (+1.87%) - utf8view_1000_scalar_single_delimiter_pos: 20.8 µs → 20.9 µs (+0.48%) - utf8view_10000_array_long_delimiter: 998.4 µs → 1025.4 µs (+2.70%) - utf8view_10000_array_single_delimiter: 414.9 µs → 415.7 µs (+0.19%) - utf8view_10000_scalar_long_delimiter_neg: 393.7 µs → 395.9 µs (+0.56%) - utf8view_10000_scalar_long_delimiter_pos: 253.4 µs → 252.7 µs (−0.28%) - utf8view_10000_scalar_single_delimiter_neg: 214.5 µs → 217.3 µs (+1.31%) - utf8view_10000_scalar_single_delimiter_pos: 207.9 µs → 208.7 µs (+0.38%) This PR doesn't touch the Utf8View code path, so the Utf8View regressions above are likely measurement noise. ## What changes are included in this PR? * Optimize `substr_index` by switching from Arrow string builders to bulk-NULL string builders ## Are these changes tested? Yes, covered by existing tests. ## Are there any user-facing changes? No. --------- Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
## Which issue does this PR close? - Related to apache#21651 from @Dandandan ## Rationale for this change While reviewing apache#21651 I found that `sin(x)` is hard coded to always return `[-1, 1]` as its bounds regardless of its input: https://github.com/apache/datafusion/blob/5901df58b21b8b4e36011744e7ddc17bcb6a37b3/datafusion/functions/src/math/bounds.rs#L27-L32 This is not clear from the docs and apache#21651 was assuming something different ## What changes are included in this PR? 1. Update the documentation to reflect the current state of the code (the bound are not exact) ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->
…che#21887) Bumps [taiki-e/install-action](https://github.com/taiki-e/install-action) from 2.75.18 to 2.75.23. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/taiki-e/install-action/releases">taiki-e/install-action's releases</a>.</em></p> <blockquote> <h2>2.75.23</h2> <ul> <li> <p>Update <code>vacuum@latest</code> to 0.26.2.</p> </li> <li> <p>Update <code>tombi@latest</code> to 0.9.24.</p> </li> <li> <p>Update <code>mise@latest</code> to 2026.4.22.</p> </li> <li> <p>Update <code>martin@latest</code> to 1.7.0.</p> </li> <li> <p>Update <code>git-cliff@latest</code> to 2.13.1.</p> </li> <li> <p>Update <code>cargo-tarpaulin@latest</code> to 0.35.4.</p> </li> <li> <p>Update <code>cargo-sort@latest</code> to 2.1.4.</p> </li> </ul> <h2>2.75.22</h2> <ul> <li> <p>Update <code>tombi@latest</code> to 0.9.22.</p> </li> <li> <p>Update <code>biome@latest</code> to 2.4.13.</p> </li> </ul> <h2>2.75.21</h2> <ul> <li> <p>Update <code>mise@latest</code> to 2026.4.19.</p> </li> <li> <p>Update <code>tombi@latest</code> to 0.9.21.</p> </li> <li> <p>Update <code>syft@latest</code> to 1.43.0.</p> </li> </ul> <h2>2.75.20</h2> <ul> <li> <p>Update <code>prek@latest</code> to 0.3.10.</p> </li> <li> <p>Update <code>cargo-xwin@latest</code> to 0.22.0.</p> </li> </ul> <h2>2.75.19</h2> <ul> <li> <p>Update <code>wasmtime@latest</code> to 44.0.0.</p> </li> <li> <p>Update <code>tombi@latest</code> to 0.9.20.</p> </li> <li> <p>Update <code>martin@latest</code> to 1.6.0.</p> </li> <li> <p>Update <code>just@latest</code> to 1.50.0.</p> </li> <li> <p>Update <code>mise@latest</code> to 2026.4.18.</p> </li> <li> <p>Update <code>rclone@latest</code> to 1.73.5.</p> </li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/taiki-e/install-action/blob/main/CHANGELOG.md">taiki-e/install-action's changelog</a>.</em></p> <blockquote> <h1>Changelog</h1> <p>All notable changes to this project will be documented in this file.</p> <p>This project adheres to <a href="https://semver.org">Semantic Versioning</a>.</p> <!-- raw HTML omitted --> <h2>[Unreleased]</h2> <ul> <li> <p>Update <code>prek@latest</code> to 0.3.11.</p> </li> <li> <p>Update <code>mise@latest</code> to 2026.4.23.</p> </li> <li> <p>Update <code>vacuum@latest</code> to 0.26.3.</p> </li> </ul> <h2>[2.75.23] - 2026-04-27</h2> <ul> <li> <p>Update <code>vacuum@latest</code> to 0.26.2.</p> </li> <li> <p>Update <code>tombi@latest</code> to 0.9.24.</p> </li> <li> <p>Update <code>mise@latest</code> to 2026.4.22.</p> </li> <li> <p>Update <code>martin@latest</code> to 1.7.0.</p> </li> <li> <p>Update <code>git-cliff@latest</code> to 2.13.1.</p> </li> <li> <p>Update <code>cargo-tarpaulin@latest</code> to 0.35.4.</p> </li> <li> <p>Update <code>cargo-sort@latest</code> to 2.1.4.</p> </li> </ul> <h2>[2.75.22] - 2026-04-25</h2> <ul> <li> <p>Update <code>tombi@latest</code> to 0.9.22.</p> </li> <li> <p>Update <code>biome@latest</code> to 2.4.13.</p> </li> </ul> <h2>[2.75.21] - 2026-04-24</h2> <ul> <li> <p>Update <code>mise@latest</code> to 2026.4.19.</p> </li> <li> <p>Update <code>tombi@latest</code> to 0.9.21.</p> </li> <li> <p>Update <code>syft@latest</code> to 1.43.0.</p> </li> </ul> <h2>[2.75.20] - 2026-04-23</h2> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/taiki-e/install-action/commit/481c34c1cf3a84c68b5e46f4eccfc82af798415a"><code>481c34c</code></a> Release 2.75.23</li> <li><a href="https://github.com/taiki-e/install-action/commit/9109405c61e64f1e88e55c82819a46c97e835f4c"><code>9109405</code></a> Update changelog</li> <li><a href="https://github.com/taiki-e/install-action/commit/cfd6b0ff1e194a5fc4fd4f692b7f06903f648672"><code>cfd6b0f</code></a> Update <code>vacuum@latest</code> to 0.26.2</li> <li><a href="https://github.com/taiki-e/install-action/commit/b2ea47bca26c10b3e7128c6d89359c70fa463469"><code>b2ea47b</code></a> Update uv manifest</li> <li><a href="https://github.com/taiki-e/install-action/commit/298078d06022bb442ff3a446a0387d1e51175e59"><code>298078d</code></a> Update <code>tombi@latest</code> to 0.9.24</li> <li><a href="https://github.com/taiki-e/install-action/commit/4c81eb55c75148bda1b398a2c8fe099617e19bec"><code>4c81eb5</code></a> Update prek manifest</li> <li><a href="https://github.com/taiki-e/install-action/commit/bb5f177651d41c019b21549ec8b706eb08ecf371"><code>bb5f177</code></a> Update <code>mise@latest</code> to 2026.4.22</li> <li><a href="https://github.com/taiki-e/install-action/commit/c630b05e422c987c36e541b211d0330d66f20bcc"><code>c630b05</code></a> Update <code>martin@latest</code> to 1.7.0</li> <li><a href="https://github.com/taiki-e/install-action/commit/b12e2ca23107113b92a8280c40d7ab6d82318049"><code>b12e2ca</code></a> Update <code>git-cliff@latest</code> to 2.13.1</li> <li><a href="https://github.com/taiki-e/install-action/commit/3a693fd6338bba43206ee4454d311fc4160872f3"><code>3a693fd</code></a> Update <code>cargo-tarpaulin@latest</code> to 0.35.4</li> <li>Additional commits viewable in <a href="https://github.com/taiki-e/install-action/compare/055f5df8c3f65ea01cd41e9dc855becd88953486...481c34c1cf3a84c68b5e46f4eccfc82af798415a">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…to >=0.17.1,<1 in /docs (apache#21889) Updates the requirements on [pydata-sphinx-theme](https://github.com/pydata/pydata-sphinx-theme) to permit the latest version. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/pydata/pydata-sphinx-theme/releases">pydata-sphinx-theme's releases</a>.</em></p> <blockquote> <h2>v0.17.1</h2> <h2>What's Changed</h2> <h3>Fixes</h3> <ul> <li>Fix docs build with sphinx 'extensions' config key by <a href="https://github.com/Yann-P"><code>@Yann-P</code></a> in <a href="https://redirect.github.com/pydata/pydata-sphinx-theme/pull/2352">pydata/pydata-sphinx-theme#2352</a></li> <li>Fix default behavior when shorten_urls is unset by <a href="https://github.com/hntk03"><code>@hntk03</code></a> in <a href="https://redirect.github.com/pydata/pydata-sphinx-theme/pull/2351">pydata/pydata-sphinx-theme#2351</a></li> <li>Fix search input receiving keystrokes after escape dismiss in Safari by <a href="https://github.com/tmchow"><code>@tmchow</code></a> in <a href="https://redirect.github.com/pydata/pydata-sphinx-theme/pull/2338">pydata/pydata-sphinx-theme#2338</a></li> <li>Replace broken Unsplash image URLs with Picsum in light-dark guide by <a href="https://github.com/tmchow"><code>@tmchow</code></a> in <a href="https://redirect.github.com/pydata/pydata-sphinx-theme/pull/2341">pydata/pydata-sphinx-theme#2341</a></li> </ul> <h3>Improvements</h3> <ul> <li>feat: add disable_search theme option by <a href="https://github.com/PhilipSchmid"><code>@PhilipSchmid</code></a> in <a href="https://redirect.github.com/pydata/pydata-sphinx-theme/pull/2315">pydata/pydata-sphinx-theme#2315</a></li> </ul> <h3>Dependencies</h3> <ul> <li>Bump pydata/pydata-sphinx-theme/.github/workflows/docs.yml from 43f9003549c495bcb43483fa21b6c1c396247f39 to cd00a86a87aba3903543c00c5ce5f009c436e0b1 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/pydata/pydata-sphinx-theme/pull/2342">pydata/pydata-sphinx-theme#2342</a></li> <li>Bump pydata/pydata-sphinx-theme/.github/workflows/CI.yml from e8db643b990df73812cf9397bc0f8cfa1164e4d3 to cd00a86a87aba3903543c00c5ce5f009c436e0b1 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/pydata/pydata-sphinx-theme/pull/2345">pydata/pydata-sphinx-theme#2345</a></li> <li>Bump octokit/request-action from 2.4.0 to 3.0.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/pydata/pydata-sphinx-theme/pull/2346">pydata/pydata-sphinx-theme#2346</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/PhilipSchmid"><code>@PhilipSchmid</code></a> made their first contribution in <a href="https://redirect.github.com/pydata/pydata-sphinx-theme/pull/2315">pydata/pydata-sphinx-theme#2315</a></li> <li><a href="https://github.com/tmchow"><code>@tmchow</code></a> made their first contribution in <a href="https://redirect.github.com/pydata/pydata-sphinx-theme/pull/2338">pydata/pydata-sphinx-theme#2338</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/pydata/pydata-sphinx-theme/compare/v0.17.0...v0.17.1">https://github.com/pydata/pydata-sphinx-theme/compare/v0.17.0...v0.17.1</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pydata/pydata-sphinx-theme/commit/a4986faa4b8241e041634d39ca6ed0e30c175240"><code>a4986fa</code></a> bump: 0.17.0 → 0.17.1 (<a href="https://redirect.github.com/pydata/pydata-sphinx-theme/issues/2355">#2355</a>)</li> <li><a href="https://github.com/pydata/pydata-sphinx-theme/commit/ff8be647971c91f759ef20222355ea57cc39362f"><code>ff8be64</code></a> Fix default behavior when shorten_urls is unset (<a href="https://redirect.github.com/pydata/pydata-sphinx-theme/issues/2351">#2351</a>)</li> <li><a href="https://github.com/pydata/pydata-sphinx-theme/commit/086775948a2e98ede34ccc0568508127eac5dc92"><code>0867759</code></a> Replace broken Unsplash image URLs with Picsum in light-dark guide (<a href="https://redirect.github.com/pydata/pydata-sphinx-theme/issues/2341">#2341</a>)</li> <li><a href="https://github.com/pydata/pydata-sphinx-theme/commit/56157e947983f5dee69650619e4c991ebe1016ba"><code>56157e9</code></a> Bump octokit/request-action from 2.4.0 to 3.0.0 (<a href="https://redirect.github.com/pydata/pydata-sphinx-theme/issues/2346">#2346</a>)</li> <li><a href="https://github.com/pydata/pydata-sphinx-theme/commit/8cce21de32c72ae5ab063cf14152ea17704b66c8"><code>8cce21d</code></a> Bump pydata/pydata-sphinx-theme/.github/workflows/CI.yml from e8db643b990df73...</li> <li><a href="https://github.com/pydata/pydata-sphinx-theme/commit/30b4830ab76f28b4e4e752ddd82e589cfef59a22"><code>30b4830</code></a> Bump pydata/pydata-sphinx-theme/.github/workflows/docs.yml from 43f9003549c49...</li> <li><a href="https://github.com/pydata/pydata-sphinx-theme/commit/6673b56f499fa54c40c2c0c828ca21ba3f0d40c1"><code>6673b56</code></a> Fix docs build with sphinx 'extensions' config key (<a href="https://redirect.github.com/pydata/pydata-sphinx-theme/issues/2352">#2352</a>)</li> <li><a href="https://github.com/pydata/pydata-sphinx-theme/commit/ebc3f1723a51750e7e3bc3b0851d7fa88a467c83"><code>ebc3f17</code></a> Fix search input receiving keystrokes after escape dismiss in Safari (<a href="https://redirect.github.com/pydata/pydata-sphinx-theme/issues/2338">#2338</a>)</li> <li><a href="https://github.com/pydata/pydata-sphinx-theme/commit/8e676945c216502bc883baf1ccd4ef70244696f7"><code>8e67694</code></a> feat: add disable_search theme option (<a href="https://redirect.github.com/pydata/pydata-sphinx-theme/issues/2315">#2315</a>)</li> <li>See full diff in <a href="https://github.com/pydata/pydata-sphinx-theme/compare/v0.17.0...v0.17.1">compare view</a></li> </ul> </details> <br /> <details> <summary>Most Recent Ignore Conditions Applied to This Pull Request</summary> | Dependency Name | Ignore Conditions | | --- | --- | | pydata-sphinx-theme | [>= 0.16.dev0, < 0.17] | </details> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [libloading](https://github.com/nagisa/rust_libloading) from 0.8.9 to 0.9.0. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/nagisa/rust_libloading/commit/dab97c569b33bd515e16637b8dedbdc696d9ec9c"><code>dab97c5</code></a> 0.9.0 release rites</li> <li><a href="https://github.com/nagisa/rust_libloading/commit/e89380446ed04ff14fe2f8adbd8cabc82ea54ad4"><code>e893804</code></a> refactor: simplify null checks, remove some iffy implementations</li> <li><a href="https://github.com/nagisa/rust_libloading/commit/4b98285041defabf87df73f57051f5d55125758f"><code>4b98285</code></a> add no-std support</li> <li>See full diff in <a href="https://github.com/nagisa/rust_libloading/compare/0.8.9...0.9.0">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> N/A ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Looking to push a PR on optimizing `array_remove` later; discovered we were lacking coverage for `array_remove_n` and `array_remove_all` variants, as well as nested types. In the process of adding them, decided to also do some cleanup of the benchmarks, specifically to generate data in a more targeted fashion suitable to what we're benchmarking in the functions (removing a specified needle from a haystack). ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> - Refactor functions used to create test data, unifying where possible - Remove f64, decimal & binary benchmarks - Add nested i64 (list<list<i64>>) benchmarks - Add remove_n & remove_all benchmarks for i64 and nested i64 ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Benchmark only changes. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> No. <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->
## Which issue does this PR close? - Closes apache#21848. ## Rationale for this change We can use the new bulk-NULL string builder APIs to reduce NULL handling overhead in `replace`. There is a further opportunity for optimization once arrow/arrow-rs#9692 lands. Benchmarks: - `replace size=1024/replace_string_ascii_single [str_len=32]`: 17.42 µs → 14.35 µs, **−17.71%** - `replace size=1024/replace_string_view [str_len=32]`: 21.23 µs → 18.89 µs, **−11.05%** - `replace size=1024/replace_string [str_len=32]`: 20.42 µs → 18.51 µs, **−9.36%** - `replace size=1024/replace_large_string [str_len=32]`: 20.83 µs → 18.42 µs, **−11.60%** - `replace size=1024/replace_string_ascii_single [str_len=128]`: 10.61 µs → 10.37 µs, **−2.22%** - `replace size=1024/replace_string_view [str_len=128]`: 28.37 µs → 28.03 µs, **−1.21%** - `replace size=1024/replace_string [str_len=128]`: 27.96 µs → 28.03 µs, **+0.25%** (noise) - `replace size=1024/replace_large_string [str_len=128]`: 28.15 µs → 27.95 µs, **−0.70%** (noise) - `replace size=4096/replace_string_ascii_single [str_len=32]`: 68.40 µs → 56.15 µs, **−17.94%** - `replace size=4096/replace_string_view [str_len=32]`: 85.95 µs → 76.84 µs, **−10.61%** - `replace size=4096/replace_string [str_len=32]`: 82.79 µs → 74.71 µs, **−9.76%** - `replace size=4096/replace_large_string [str_len=32]`: 85.32 µs → 75.14 µs, **−11.94%** - `replace size=4096/replace_string_ascii_single [str_len=128]`: 38.35 µs → 39.10 µs, **+1.96%** (small regression) - `replace size=4096/replace_string_view [str_len=128]`: 132.77 µs → 128.08 µs, **−3.53%** - `replace size=4096/replace_string [str_len=128]`: 127.71 µs → 128.29 µs, **+0.46%** (noise) - `replace size=4096/replace_large_string [str_len=128]`: 131.97 µs → 128.68 µs, **−2.49%** ## What changes are included in this PR? * Use bulk-NULL string builders in `replace` ## Are these changes tested? Yes, covered by existing tests. ## Are there any user-facing changes? No.
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #. Followup on apache#21679 (comment) ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Raz Luvaton <16746759+rluvaton@users.noreply.github.com>
## Which issue does this PR close? - Closes apache#21901 ## Rationale for this change Get a clean CI ## What changes are included in this PR? Run ```shell cargo update -p astral-tokio-tar ``` And then check in the results ## Are these changes tested? By CI ## Are there any user-facing changes? No (and this library is used in testing, not in DataFusion)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Part of apache#21777 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> First step in supporting `ListView` types in UDFs. Previously the UDFs would error if trying to pass a `ListView` type to nested UDFs. Now we can pass in `ListView` inputs and they are cast to `List`/`LargeList` (similar to `FixedSizeList` if that coercion mode is enabled). ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> In type coercion, for array signatures automatically coerce any `ListView`/`LargeListView` types to `List`/`LargeList` respectively. Also add `ListView` support to various array util functions. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Added SLTs. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> No. <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> After apache#19759, all the methods of a `MemoryReservation` take &self instead of &mut self, so no need to hide a `SharedMemoryReservation` behind a Mutex. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> Remove an unnecessary Mutex ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> By existing unit tests ## Are there any user-facing changes? No <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->
…atch (apache#22852) ## Which issue does this PR close? - Closes apache#22849 - A related cross-partition starvation case is tracked separately in apache#22874 and addressed by an upcoming follow-up PR — see [discussion](apache#22852 (comment)) for details ## Rationale for this change `TopK::insert_batch` short-circuits when the heap's dynamic filter rejects every row in a batch: ```rust if !filter.has_true() { // nothing to filter, so no need to update return Ok(()); } ``` The early-exit check `attempt_early_completion(&batch)` lives later in the same function, gated on `replacements > 0`. So a batch that the filter rejects entirely bypasses the check. The heap's dynamic filter is derived from the heap's worst row (via `update_filter`). A batch whose rows all come from a strictly worse sort prefix is exactly the batch the filter rejects entirely — i.e. the very signal `attempt_early_completion` is designed to detect ("the next batch is past the heap's boundary, we can stop") is what causes the function to short-circuit *before* the check runs. This is a feature-interaction regression between two PRs that were both correct in isolation. The `attempt_early_completion` mechanism was added by apache#15563 (closing apache#15529). At the time, there was no heap-derived dynamic filter on TopK, so the only sensible call site was right after a successful heap insertion. Two months later, apache#15770 added the dynamic-filter pushdown for TopK sorts, introducing the `!filter.has_true()` short-circuit. The two features address different problems and the new short-circuit didn't connect to the existing prefix-completion check — which is how this gap opened up. **Consequence**: on a TopK over an input ordered on the sort prefix, `finished = true` is never set once the heap stabilizes. Since `finished` is the signal `SortExec` uses to stop pulling from its input (via `Poll::Ready(None)` from the TopK stream, which cascades into dropping the source stream), the source keeps being polled long past the point where no further row can improve the heap. The LIMIT optimization effectively degrades to "heap saves memory but reads everything"; sources with cancellable streams (e.g. networked sources) never receive the cancellation signal. ## What changes are included in this PR? Single behavioral change in `datafusion/physical-plan/src/topk/mod.rs`: call `attempt_early_completion(&batch)` immediately before the `return Ok(())` in the `!filter.has_true()` branch. Why this scope, not a broader restructuring: - The existing `attempt_early_completion` call inside `if replacements > 0` is load-bearing for a related case: a batch containing a mix of "still valuable" rows and "past the boundary" rows. The existing `test_try_finish_marks_finished_with_prefix` test covers this case — Batch 2 with `a=[2,3], b=[10,20]` against a heap where `heap.max.a = 2`; the `(2, 10)` row must be inserted before the check on the `(3, 20)` last row triggers. Moving the call earlier would skip the insertion of valuable rows and break that test. - The bug is specifically that the *short-circuit* path doesn't call the check. The fix targets exactly that path. - A related but separate gap is not addressed here: when `filter.has_true() == true` but `replacements == 0` (the filter accepts some rows but `find_new_topk_items` ends up inserting none of them), the existing call inside `if replacements > 0` is also skipped. This requires a divergence between the heap's filter predicate and the row-byte comparison used inside `find_new_topk_items`, which shouldn't normally happen (the filter is derived from the heap's worst row using the same comparator). A deterministic synthetic repro would likely require concurrent heap updates from sibling partitions or boundary-value edge cases (NaN/NULL semantics, type coercion). Happy to send a follow-up if reviewers want it covered; the workload that motivated this fix was the filter-rejection case empirically. ## Are these changes tested? Yes. Added a regression test `test_try_finish_fires_when_filter_rejects_entire_batch`. The assertion target is `topk.finished` — the flag that signals "stop pulling from the source" to upstream consumers (read by `TopKExec::poll_next` to emit `Poll::Ready(None)`). Asserting that the flag transitions on the fully-filter-rejected batch is equivalent to asserting that the source-stopping mechanism activates. - Builds a TopK over a `(a, b)` sort with prefix `a`, k=3. - Inserts a batch that fills the heap with rows from `a ∈ {1, 2}`; `update_filter` tightens the filter to `a < 2 OR (a = 2 AND b < 30)`. - Inserts a second batch with all rows at `a = 3` — filter rejects every row. - Without the fix: `insert_batch` short-circuits, `topk.finished` stays `false`. Test fails. - With the fix: `attempt_early_completion` fires (last-row prefix `a = 3` > heap.max prefix `a = 2`), `topk.finished` becomes `true`. Test passes. The test also asserts the emitted top-K is unchanged from after batch 1, confirming no candidate row was incorrectly excluded by the early bail. All 28 existing `topk::` tests continue to pass (including `test_try_finish_marks_finished_with_prefix`, which exercises the mixed-prefix case). ## Are there any user-facing changes? No public API or output changes. The fix only changes when TopK marks itself `finished = true` — specifically, it now fires `attempt_early_completion` for batches that are entirely rejected by the heap's dynamic filter, where previously it would silently skip the check. Output of TopK is unchanged; only the early-exit behavior improves. --------- Co-authored-by: Gabriel <45515538+gabotechs@users.noreply.github.com> (cherry picked from commit 6520315)
…r-22852-branch54-20260617 [branch-54] Cherry-pick apache#22852 Co-authored-by: ajegou <arnaud.jegou@gmail.com> Co-authored-by: arnaud.jegou <arnaud.jegou@datadoghq.com>
Physical plan proto serialization was still binding the plan as dyn Any after the as_any removal. That bypassed ExecutionPlan::downcast_ref, so wrapper plans that delegate their public downcast identity fell through to the extension codec instead of serializing as the wrapped built-in plan. Bind the serializer view as dyn ExecutionPlan so the existing downcast chain uses the delegating helper, and add a regression test with a wrapper around EmptyExec. (cherry picked from commit 401d8fc)
…elegate fix(proto): honor ExecutionPlan downcast_delegate during serialization
Cherry pick 434957e - turn off submodule updating
Cherry pick a0763db - Fix ArrayCompact incompatibility
Cherry pick 6692f6f - fix(substrait): dedupe names
Cherry pick 12d6c81 - Add lambda substrait support (apache#21193) (#134)
Cherry pick b502200 - fix: make PushDownLeafProjections work with unnest
Cherry pick 93ba65a - add downcast_delegate
…it (apache#23277) ## Which issue does this PR close? - Closes apache#23219. ## Rationale for this change The query from the issue: ```sql SELECT (((Cast(id AS BIGINT) % 1024) + 1024) % 1024) AS computed_bucket FROM profile ORDER BY computed_bucket, Cast(id AS BIGINT) limit 10; ``` panics: ``` thread 'main' panicked at .../datafusion-datasource-54.0.0/src/statistics.rs:100:48: index out of bounds: the len is 0 but the index is 0 ``` The underlying issue is that the current code panics when files are split by statistics and there are no statistics available for the column where the sort order is defined in this case `computed_bucket`. ## What changes are included in this PR? - Fix in `MinMaxStatistics` to check if there are stats available for a given column - Test ## Are these changes tested? Yes ## Are there any user-facing changes? No
Cherry pick 7b1bffa - fix: Avoid panicing when stats are not available for a file group spl…
* chore: Update to arrow/parquet 59.1.0 (apache#23312) (DRAFT until arrow is updated, I am using this PR to pre-test the release) - related to apache/arrow-rs#9878 Update to latest arrow Yes by CI No API change (this is a minor update of Arrow) (cherry picked from commit f34a676) * chore: Update to arrow/parquet 59.0.0 (apache#22744) - related to apache/arrow-rs#9110 Update to latest version of arrow/parquet 1. Update to arrow/parquet 59.0.0 2. Adjust code for API differences By CI New dependency (cherry picked from commit dae03ee) --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
## Which issue does this PR close? Part of apache#21172 ## Rationale for this change Protobuf support wasn't implemented in main lambda PR to not make it even bigger ## What changes are included in this PR? Protobuf encoding and decoding (~1000 LOC in generated files, ~210 impl, ~400 tests) ## Are these changes tested? Unit tests, similar to the existing ones for scalar functions ## Are there any user-facing changes? Proto `ExprType` has new variants
Update the pre-refactor upstream implementation for the concrete HigherOrderUDF wrapper already present on branch-54. Preserve the DynamicFilterPhysicalExpr import required by existing branch-54 tests and regenerate protobuf JSON bindings with the branch-54 generator.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
array_agg()performance apache/datafusion#23715.Rationale for this change
Queries using
array_agg(DISTINCT col)were significantly slower than expected.Profiling revealed that
DistinctArrayAggAccumulator::update_batchwas allocating aheap-owned
Stringon every single input row — even for rows whose value was alreadypresent in the accumulator. For a typical low-cardinality workload (e.g. a column of ~25
database names across thousands of rows), this meant paying the full allocation cost for
every duplicate, which dominated the runtime.
What changes are included in this PR?
This PR applies the same deduplication strategy already used by
AggregateExecforGROUP BY: duplicate rows now cost only a hash probe with no heap allocation, and newdistinct values are appended to a single shared buffer rather than allocated individually.
The fix applies to all column types, not just strings, and
retract_batchsupport(required for sliding window frames such as
ROWS BETWEEN N PRECEDING AND CURRENT ROW)is fully preserved.
Are these changes tested?
Four unit tests were added to
DistinctArrayAggAccumulator— one each forUtf8,Int64,Float64, andDate32— to pin the deduplication contract across the mostcommon column types and serve as a regression guard for future changes. The existing
sliding window sqllogictest suite (
array_agg_sliding_window.slt) coversretract_batchcorrectness end-to-end and passes unchanged.
Two
update_batchmicro-benchmarks were added to measure the before/after on realisticdata: one with low cardinality (~25 distinct database names in 8 192 rows, modelling the
common production case) and one with high cardinality (~7 800 distinct values, modelling
the worst case where almost every row is new). Results on an 8 192-row batch:
Are there any user-facing changes?
No user-facing changes
No breaking changes to public APIs