Skip to content

Branch 54 - #163

Closed
gene-bordegaray wants to merge 3557 commits into
mainfrom
branch-54
Closed

Branch 54#163
gene-bordegaray wants to merge 3557 commits into
mainfrom
branch-54

Conversation

@gene-bordegaray

@gene-bordegaray gene-bordegaray commented Aug 1, 2026

Copy link
Copy Markdown

Which issue does this PR close?

  • Closes #.

Rationale for this change

cherry picks range partitioning commits

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

jx2lee and others added 30 commits April 27, 2026 14:18
@jx2lee @alamb
docs: refresh CLI usage output in the user guide (apache#21874)
68061a5 
## Which issue does this PR close?

- None. This is a small documentation sync update.

## Rationale for this change

The CLI help output shown in the user guide had drifted from the current
`datafusion-cli --help` output, which makes the documentation misleading
for users trying to discover supported flags.

## What changes are included in this PR?

- Update the `datafusion-cli --help` snippet in
`docs/source/user-guide/cli/usage.md`
- Remove outdated options and defaults
- Add newer flags and current help text formatting

## Are these changes tested?

- Not with automated tests. This is a documentation-only change that
updates the checked-in help output.

## Are there any user-facing changes?

- Yes. The user guide now matches the current CLI usage output more
closely.

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
@alamb
docs: clarify ExecutionProps and TaskContext docs (apache#21872)
bff0ffb 
## Which issue does this PR close?

- Follow on to apache#21240

## Rationale for this change

While reviewing apache#21240 I found
that the existing doc comments describing the relationship between
ExecutionProps and TaskContext were vague

## What changes are included in this PR?

- Update the doc comments on ExecutionProps and TaskContext
- Clarify why the two structures remain separate

## Are these changes tested?

Not with automated tests. This is a comment-only change.

## Are there any user-facing changes?

Docs only
@Geethapranay1 @comphead
chore: add internal markdown link check (apache#21831)
2676297 
## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->

- Closes apache#21747 

## Rationale for this change
datafusion did not have a CI check for broken links in markdown content,
docs workflows build and deploy docs, and dev checks formatting and
spelling, but none of them validate link targets.
This pr adds a dedicated link check for internal markdown links so
broken references fail early in PRs.
I kept the scope internal-only to avoid flaky CI failures from external
websites and rate limits.
Rust doc comments remain covered by the existing rustdoc CI job.



<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->

## What changes are included in this PR?


- Added a new Dev workflow job, **Check Markdown Links**, in `dev.yml`.
- Added `LYCHEE_VERSION` pin in `tool_versions.sh`.
- Added `markdown_link_check.sh` to run lychee on the selected markdown
paths.
- Added `lychee.toml` with internal-link policy and exclusions.
- Added check markdown links to required status checks in `.asf.yaml`.
- Updated contributor testing docs with the new local command and scope
note.
- Fixed internal markdown links that failed under the new check in:
  - `roadmap.md`
  - `49.0.0.md`
  - `overview.md`
  - `dataframe.md`
  - `format_options.md`
<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

## Are these changes tested?
Yes,

- `python3 ci/scripts/check_asf_yaml_status_checks.py` passed.
- `bash -n ci/scripts/markdown_link_check.sh` passed.
- `bash ci/scripts/markdown_link_check.sh` passed with 0 errors.
- `cargo fmt --all --check` passed.

OK: All 5 required_status_checks match existing GitHub Actions jobs.
🔍 12824 Total (in 0s) ✅ 490 OK 🚫 0 Errors 👻 12334 Excluded
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

## Are there any user-facing changes?
No,

There is one contributor-facing CI change: PRs now fail when internal
markdown links break in the checked markdown files.

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->

---------

Co-authored-by: Oleks V <comphead@users.noreply.github.com>
@neilconway @alamb
perf: Use bulk-NULL builder in chr (apache#21847)
62ad66b 
## Which issue does this PR close?

- Closes apache#21846.

## Rationale for this change

Optimize `chr` by avoiding per-row NULL bitmap maintenance, and also
split the hot loop to avoid taking a branch when no NULL bitmap is
given.

Benchmarks:

```
chr/array: 3.8768 µs → 3.1548 µs, −18.57% (`p < 0.05`)
```

## What changes are included in this PR?

* Optimize `chr` to reduce NULL-handling overhead

## Are these changes tested?

Yes.

## Are there any user-facing changes?

No.

---------

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
@akoshchiy @alamb
feat: remove __unnest_placeholder from struct unnest projection (ap…
40b209e 
…ache#21725)

## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->

- Closes apache#16894 

## What changes are included in this PR?

- User-visible struct-UNNEST columns no longer expose
`__unnest_placeholder`.
- Internal Unnest planning still uses placeholders.
- This adds a final Projection layer in explain plans wherever struct
UNNEST output is published.

<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

## Are these changes tested?
Yes.
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

## Are there any user-facing changes?
Yes. The aliases of unnested columns will be changed.
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->

---------

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
@azhangd @alamb
perf: implement convert_to_state for SparkAvg (apache#21548)
ca1d39d 
## Which issue does this PR close?

- Part of apache#17964.

## Rationale for this change

SparkAvg's AvgGroupsAccumulator doesn't implement
supports_convert_to_state (defaults to false), which prevents the
skip-partial-aggregation optimization from kicking in for queries that
use Spark's avg().

I ran into this while benchmarking a Spark Connect engine built on
DataFusion. On TPC-H q17 at SF10, the partial aggregate for
avg(l_quantity) grouped by l_partkey (~2M groups out of 60M rows) was
not triggering skip-aggregation:

| Metric | Without convert_to_state | With convert_to_state |
|--------|-------------------------|-----------------------|
| Partial aggregate memory | 923 MB | 40 MB |
| Partial aggregate elapsed | 4.75s | 109ms |

The skip-aggregation probe (apache#11627) detects when a partial aggregate
isn't reducing cardinality and falls back to passing rows through as
state directly. This needs convert_to_state so the accumulator can
produce [sum, count] state arrays from raw input. The built-in Avg
already has this (apache#11734), but it wasn't carried over when SparkAvg was
migrated from Comet in apache#17871.

## What changes are included in this PR?

Adds convert_to_state() and supports_convert_to_state() to
AvgGroupsAccumulator in datafusion-spark.

Follows the same approach as the built-in Avg, adapted for SparkAvg's
differences:
- State order is [sum, count] (vs [count, sum] in the built-in)
- Count type is Int64 (vs UInt64 in the built-in)
- Null handling uses NullBuffer::union directly instead of pulling in
datafusion-functions-aggregate-common as a dep

Also cleaned up the fully-qualified arrow::array::BooleanArray
references in update_batch / merge_batch since adding BooleanArray to
the import block triggered the unused_qualifications lint.

## Are these changes tested?

Yes, unit tests covering basic conversion, null propagation, filter
handling, and a roundtrip through merge_batch to verify the converted
state produces correct results end-to-end.

## Are there any user-facing changes?

No. Queries using avg() through the Spark function registry will
automatically benefit from skip-partial-aggregation on high-cardinality
groupings.

---------

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
@neilconway @alamb
perf: Implement physical execution of uncorrelated scalar subqueries (a…
1bb588e 
…pache#21240)

## Which issue does this PR close?

- Closes apache#3781.
- Closes apache#18181.

## Rationale for this change

Previously, DataFusion evaluated uncorrelated scalar subqueries by
transforming them into joins. This has three shortcomings:

1. Scalar subqueries that return > 1 row were allowed, producing
incorrect query results. Such queries should instead result in a runtime
error.
2. Performance. Evaluating scalar subqueries as a join requires going
through the join machinery. More importantly, it means that UDFs that
have specialized handling of scalar inputs cannot use those code paths
for scalar subqueries, which often results in significantly slower query
execution (e.g., apache#18181). It also makes filter pushdown for scalar
subquery filters more difficult (apache#21324)
3. Uncorrelated scalar subqueries previously did not work in `ORDER BY`
or `JOIN ON`, or as arguments to an aggregate function. Those cases are
now supported.

This PR introduces physical execution of uncorrelated scalar subqueries:

* Uncorrelated subqueries are left in the plan by the optimizer, not
rewritten into joins
* The physical planner collects uncorrelated scalar subqueries and plans
them recursively (supporting nested subqueries). We add a
`ScalarSubqueryExec` plan node to the top of any physical plan with
uncorrelated subqueries: it has N+1 children, N subqueries and its
"main" input, which is the rest of the query plan. The subquery
expression in the parent plan is replaced with a `ScalarSubqueryExpr`.
* `ScalarSubqueryExec` manages the execution of the subqueries. Subquery
evaluation is done in parallel (for a given query level), but at present
it happens strictly before evaluation of the parent query. This might be
improved in the future (apache#21591).
* `ScalarSubqueryExpr` reads its value from a shared slot that
`ScalarSubqueryExec` populates when the subquery finishes; the physical
planner assigns each subquery its slot index via `ExecutionProps`.

This architecture makes it easy to avoid the shortcomings described
above. Performance seems roughly unchanged (benchmarks added in this
PR), but in situations like apache#18181, we can now leverage scalar
fast-paths; in the case of apache#18181 specifically, this improves
performance from ~800 ms to ~30 ms.

## What changes are included in this PR?

* Modify subquery rewriter to not transform subqueries -> joins
* Collect and plan uncorrelated scalar subqueries in the physical
planner, and wire up `ScalarSubqueryExpr`
* Support for subqueries in physical plan serialization/deserialization
using `PhysicalProtoConverterExtension` to wire up `ScalarSubqueryExpr`
correctly
* Support for subqueries in logical plan serialization/deserialization
* Add various SLT tests and update expected plan shapes for some tests

## Are these changes tested?

Yes. New SLT coverage for cardinality errors, `ORDER BY` / `JOIN ON` /
aggregate-arg contexts, nested uncorrelated subqueries,
duplicate-subquery deduplication, and partition-pruning filters; new
roundtrip tests for logical and physical plan serialization.

## Are there any user-facing changes?

SQL:
* Uncorrelated scalar subqueries that return more than one row now
result in a runtime error, instead of silently producing incorrect
results.
* Uncorrelated scalar subqueries now work in `ORDER BY`, `JOIN ON`, and
as aggregate function arguments.

Rust APIs:
* In `datafusion-proto`, breaking changes to
`Serializeable::from_bytes_with_registry` (renamed to
`from_bytes_with_ctx`), `parse_expr` / `parse_sorts` / `parse_exprs`,
and the `PhysicalProtoConverterExtension` trait.

Plan shape:
* `LogicalPlan::Subquery` nodes will now be preserved in the logical
plan
* Physical plans can now contain `ScalarSubqueryExec` plan node and
`ScalarSubqueryExpr` expressions

The wire format has also changed to include scalar subqueries.

---------

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
@theirix @alamb
perf: optimise first_value, last_value aggregate function (apache…
95ef332 
…#21383)

## Which issue does this PR close?

- Refers to apache#17899

## Rationale for this change

A performance improvement of `first_last.rs` (up to 36%), based on the
original author's observations.

UPD: rebased on yesterday's refactoring in apache#17899 - сс @UBarney @alamb 

## What changes are included in this PR?

- Optimise memory allocation in `take_state` - no need to copy vectors
and buffers
- Optimise extracting single elements
- Pre-compute common data for sorting
~~- Rename structs and functions as recommended in TODOs - a majority of
changes in this PR~~
- Add benchmark. It's pretty complicated to test aggregates with
grouping, since many operations are stateful, so I introduced end-to-end
`evaluate` test (to actually test taking state) and `convert_to_state`
(as in other benches)

## Are these changes tested?

- Existing unit and integration tests
- A new bench has a meaningful result

Improvements: up to 36%

Raw bench result:
<details>
first_value convert_to_state nulls=0%, filter=false
                        time:   [98.086 µs 99.014 µs 100.25 µs]
change: [−15.400% −13.532% −11.559%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 17 outliers among 100 measurements (17.00%)
  4 (4.00%) low mild
  3 (3.00%) high mild
  10 (10.00%) high severe

Benchmarking first_value evaluate_bench nulls=0%, filter=false,
first(2): Collecting 100 samples in estimated 7.1575 s (10k ite
first_value evaluate_bench nulls=0%, filter=false, first(2)
                        time:   [54.938 µs 55.482 µs 56.060 µs]
change: [−38.163% −36.295% −34.463%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
  5 (5.00%) high mild
  4 (4.00%) high severe

Benchmarking first_value evaluate_bench nulls=0%, filter=false, all:
Collecting 100 samples in estimated 7.0741 s (10k iteratio
first_value evaluate_bench nulls=0%, filter=false, all
                        time:   [50.624 µs 51.092 µs 51.612 µs]
change: [−17.955% −16.332% −14.593%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
  5 (5.00%) high mild
  4 (4.00%) high severe

first_value convert_to_state nulls=0%, filter=true
                        time:   [2.0647 µs 2.0881 µs 2.1148 µs]
change: [−7.6314% −6.2174% −4.8530%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  2 (2.00%) high mild
  6 (6.00%) high severe

Benchmarking first_value evaluate_bench nulls=0%, filter=true, first(2):
Collecting 100 samples in estimated 9.6354 s (10k iter
first_value evaluate_bench nulls=0%, filter=true, first(2)
                        time:   [54.708 µs 55.240 µs 55.805 µs]
change: [−27.249% −24.755% −22.195%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  3 (3.00%) high mild
  2 (2.00%) high severe

Benchmarking first_value evaluate_bench nulls=0%, filter=true, all:
Collecting 100 samples in estimated 9.6394 s (10k iteration
first_value evaluate_bench nulls=0%, filter=true, all
                        time:   [50.540 µs 50.963 µs 51.424 µs]
change: [−6.8346% −3.9146% −0.8864%] (p = 0.01 < 0.05)
                        Change within noise threshold.
Found 4 outliers among 100 measurements (4.00%)
  1 (1.00%) high mild
  3 (3.00%) high severe

Benchmarking first_value convert_to_state nulls=90%, filter=false:
Collecting 100 samples in estimated 5.0440 s (50k iterations
first_value convert_to_state nulls=90%, filter=false
                        time:   [98.054 µs 98.996 µs 100.10 µs]
change: [−3.4798% −2.2129% −1.0173%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 19 outliers among 100 measurements (19.00%)
  1 (1.00%) low severe
  8 (8.00%) low mild
  2 (2.00%) high mild
  8 (8.00%) high severe

Benchmarking first_value evaluate_bench nulls=90%, filter=false,
first(2): Collecting 100 samples in estimated 8.5385 s (10k it
first_value evaluate_bench nulls=90%, filter=false, first(2)
                        time:   [53.780 µs 54.673 µs 55.639 µs]
change: [−17.702% −15.978% −14.173%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  1 (1.00%) low mild
  5 (5.00%) high mild
  2 (2.00%) high severe

Benchmarking first_value evaluate_bench nulls=90%, filter=false, all:
Collecting 100 samples in estimated 8.2692 s (10k iterati
first_value evaluate_bench nulls=90%, filter=false, all
                        time:   [49.851 µs 50.289 µs 50.755 µs]
change: [−4.8554% −3.1896% −1.3951%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  3 (3.00%) low severe
  2 (2.00%) high mild
  3 (3.00%) high severe

Benchmarking first_value convert_to_state nulls=90%, filter=true:
Collecting 100 samples in estimated 5.0077 s (2.4M iterations
first_value convert_to_state nulls=90%, filter=true
                        time:   [2.0339 µs 2.0465 µs 2.0603 µs]
change: [−1.8037% −0.7068% +0.3821%] (p = 0.22 > 0.05)
                        No change in performance detected.
Found 8 outliers among 100 measurements (8.00%)
  2 (2.00%) low mild
  3 (3.00%) high mild
  3 (3.00%) high severe

Benchmarking first_value evaluate_bench nulls=90%, filter=true,
first(2): Collecting 100 samples in estimated 9.7925 s (10k ite
first_value evaluate_bench nulls=90%, filter=true, first(2)
                        time:   [54.544 µs 55.119 µs 55.720 µs]
change: [−15.717% −13.982% −12.279%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) low mild
  1 (1.00%) high severe

Benchmarking first_value evaluate_bench nulls=90%, filter=true, all:
Collecting 100 samples in estimated 9.7401 s (10k iteratio
first_value evaluate_bench nulls=90%, filter=true, all
                        time:   [50.126 µs 50.886 µs 51.703 µs]
change: [+0.8379% +4.1713% +7.1024%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 9 outliers among 100 measurements (9.00%)
  6 (6.00%) high mild
  3 (3.00%) high severe

last_value convert_to_state nulls=0%, filter=false
                        time:   [97.957 µs 98.314 µs 98.692 µs]
change: [−2.8086% −2.0315% −1.2541%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
  7 (7.00%) high mild
  3 (3.00%) high severe

Benchmarking last_value evaluate_bench nulls=0%, filter=false, first(2):
Collecting 100 samples in estimated 7.0582 s (10k iter
last_value evaluate_bench nulls=0%, filter=false, first(2)
                        time:   [52.692 µs 53.414 µs 54.144 µs]
change: [−22.228% −20.636% −19.153%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  1 (1.00%) low mild
  2 (2.00%) high mild
  2 (2.00%) high severe

Benchmarking last_value evaluate_bench nulls=0%, filter=false, all:
Collecting 100 samples in estimated 6.9411 s (10k iteration
last_value evaluate_bench nulls=0%, filter=false, all
                        time:   [49.781 µs 50.226 µs 50.793 µs]
change: [−1.9658% −0.1634% +1.5825%] (p = 0.86 > 0.05)
                        No change in performance detected.
Found 11 outliers among 100 measurements (11.00%)
  1 (1.00%) low mild
  6 (6.00%) high mild
  4 (4.00%) high severe

last_value convert_to_state nulls=0%, filter=true
                        time:   [2.0639 µs 2.0781 µs 2.0949 µs]
change: [−0.7535% +0.4491% +1.6752%] (p = 0.47 > 0.05)
                        No change in performance detected.
Found 11 outliers among 100 measurements (11.00%)
  3 (3.00%) low mild
  4 (4.00%) high mild
  4 (4.00%) high severe

Benchmarking last_value evaluate_bench nulls=0%, filter=true, first(2):
Collecting 100 samples in estimated 9.8040 s (10k itera
last_value evaluate_bench nulls=0%, filter=true, first(2)
                        time:   [53.779 µs 54.311 µs 54.868 µs]
change: [−15.863% −14.071% −12.391%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  4 (4.00%) high mild
  3 (3.00%) high severe

Benchmarking last_value evaluate_bench nulls=0%, filter=true, all:
Collecting 100 samples in estimated 9.6860 s (10k iterations
last_value evaluate_bench nulls=0%, filter=true, all
                        time:   [50.276 µs 50.794 µs 51.429 µs]
change: [−1.6780% +0.0697% +1.9541%] (p = 0.94 > 0.05)
                        No change in performance detected.
Found 6 outliers among 100 measurements (6.00%)
  2 (2.00%) high mild
  4 (4.00%) high severe

last_value convert_to_state nulls=90%, filter=false
                        time:   [97.508 µs 98.412 µs 99.486 µs]
change: [−1.2994% −0.3404% +0.8319%] (p = 0.52 > 0.05)
                        No change in performance detected.
Found 9 outliers among 100 measurements (9.00%)
  2 (2.00%) low mild
  3 (3.00%) high mild
  4 (4.00%) high severe

Benchmarking last_value evaluate_bench nulls=90%, filter=false,
first(2): Collecting 100 samples in estimated 8.9282 s (10k ite
last_value evaluate_bench nulls=90%, filter=false, first(2)
                        time:   [54.064 µs 54.748 µs 55.433 µs]
change: [−15.790% −13.678% −11.790%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  3 (3.00%) low mild
  1 (1.00%) high mild
  1 (1.00%) high severe

Benchmarking last_value evaluate_bench nulls=90%, filter=false, all:
Collecting 100 samples in estimated 9.2411 s (10k iteratio
last_value evaluate_bench nulls=90%, filter=false, all
                        time:   [49.964 µs 50.731 µs 51.630 µs]
change: [−2.4530% −0.2470% +1.8407%] (p = 0.82 > 0.05)
                        No change in performance detected.
Found 9 outliers among 100 measurements (9.00%)
  8 (8.00%) high mild
  1 (1.00%) high severe

last_value convert_to_state nulls=90%, filter=true
                        time:   [2.0660 µs 2.0874 µs 2.1139 µs]
change: [−3.2299% −1.8585% −0.5850%] (p = 0.01 < 0.05)
                        Change within noise threshold.
Found 10 outliers among 100 measurements (10.00%)
  4 (4.00%) high mild
  6 (6.00%) high severe

Benchmarking last_value evaluate_bench nulls=90%, filter=true, first(2):
Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to
increase target time to 5.6s, enable flat sampling, or reduce sample
count to 60.
Benchmarking last_value evaluate_bench nulls=90%, filter=true, first(2):
Collecting 100 samples in estimated 5.6274 s (5050 ite
last_value evaluate_bench nulls=90%, filter=true, first(2)
                        time:   [53.565 µs 54.399 µs 55.381 µs]
change: [−18.757% −16.680% −14.334%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  8 (8.00%) high mild

Benchmarking last_value evaluate_bench nulls=90%, filter=true, all:
Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to
increase target time to 5.3s, enable flat sampling, or reduce sample
count to 60.
Benchmarking last_value evaluate_bench nulls=90%, filter=true, all:
Collecting 100 samples in estimated 5.3402 s (5050 iteratio
last_value evaluate_bench nulls=90%, filter=true, all
                        time:   [49.473 µs 50.055 µs 50.743 µs]
change: [−41.448% −32.836% −24.471%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
  5 (5.00%) high mild
  4 (4.00%) high severe

cargo bench --bench first_last -- --baseline main-first_last3 810.16s
user 14.72s system 135% cpu 10:09.30 total
irix@tau ~/projects/third-party/datafusion [optimise-first_last
(theirix/optimise-first_last)?] % time cargo bench --bench first_last --
--baseline main-first_last4
Compiling datafusion-functions-aggregate v53.0.0
(/Users/irix/projects/third-party/datafusion/datafusion/functions-aggregate)
    Finished `bench` profile [optimized] target(s) in 5m 04s
Running benches/first_last.rs
(target/release/deps/first_last-c2812404ab64de23)
Gnuplot not found, using plotters backend
first_value convert_to_state nulls=0%, filter=false
                        time:   [100.29 µs 100.92 µs 101.62 µs]
change: [−1.3141% −0.4446% +0.4530%] (p = 0.34 > 0.05)
                        No change in performance detected.
Found 9 outliers among 100 measurements (9.00%)
  7 (7.00%) high mild
  2 (2.00%) high severe

Benchmarking first_value evaluate_bench nulls=0%, filter=false,
first(2): Collecting 100 samples in estimated 7.6058 s (10k ite
first_value evaluate_bench nulls=0%, filter=false, first(2)
                        time:   [57.325 µs 58.842 µs 60.567 µs]
change: [−14.545% −12.765% −10.817%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  3 (3.00%) high mild
  2 (2.00%) high severe

Benchmarking first_value evaluate_bench nulls=0%, filter=false, all:
Collecting 100 samples in estimated 7.1421 s (10k iteratio
first_value evaluate_bench nulls=0%, filter=false, all
                        time:   [52.632 µs 53.182 µs 53.814 µs]
change: [−9.5351% −6.9990% −4.3937%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
  1 (1.00%) low mild
  6 (6.00%) high mild
  3 (3.00%) high severe

first_value convert_to_state nulls=0%, filter=true
                        time:   [2.2486 µs 2.3289 µs 2.4224 µs]
change: [−14.956% −12.528% −9.6570%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 19 outliers among 100 measurements (19.00%)
  2 (2.00%) high mild
  17 (17.00%) high severe

Benchmarking first_value evaluate_bench nulls=0%, filter=true, first(2):
Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to
increase target time to 5.3s, enable flat sampling, or reduce sample
count to 60.
Benchmarking first_value evaluate_bench nulls=0%, filter=true, first(2):
Collecting 100 samples in estimated 5.3276 s (5050 ite
first_value evaluate_bench nulls=0%, filter=true, first(2)
                        time:   [60.696 µs 62.295 µs 63.977 µs]
change: [−28.986% −27.090% −25.120%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  2 (2.00%) low mild
  5 (5.00%) high mild

Benchmarking first_value evaluate_bench nulls=0%, filter=true, all:
Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to
increase target time to 5.1s, enable flat sampling, or reduce sample
count to 60.
Benchmarking first_value evaluate_bench nulls=0%, filter=true, all:
Collecting 100 samples in estimated 5.1224 s (5050 iteratio
first_value evaluate_bench nulls=0%, filter=true, all
                        time:   [53.834 µs 54.827 µs 55.913 µs]
change: [−12.918% −8.8195% −4.0336%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  3 (3.00%) low mild
  1 (1.00%) high mild
  3 (3.00%) high severe

Benchmarking first_value convert_to_state nulls=90%, filter=false:
Collecting 100 samples in estimated 5.1237 s (50k iterations
first_value convert_to_state nulls=90%, filter=false
                        time:   [104.46 µs 106.26 µs 108.42 µs]
change: [−11.589% −8.9599% −6.3106%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
  6 (6.00%) high mild

Benchmarking first_value evaluate_bench nulls=90%, filter=false,
first(2): Collecting 100 samples in estimated 8.1885 s (10k it
first_value evaluate_bench nulls=90%, filter=false, first(2)
                        time:   [59.528 µs 60.651 µs 61.809 µs]
change: [−21.550% −19.288% −16.929%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  5 (5.00%) high mild

Benchmarking first_value evaluate_bench nulls=90%, filter=false, all:
Collecting 100 samples in estimated 8.2091 s (10k iterati
first_value evaluate_bench nulls=90%, filter=false, all
                        time:   [53.384 µs 54.251 µs 55.248 µs]
change: [−9.1283% −6.6250% −4.4009%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 13 outliers among 100 measurements (13.00%)
  7 (7.00%) high mild
  6 (6.00%) high severe

Benchmarking first_value convert_to_state nulls=90%, filter=true:
Collecting 100 samples in estimated 5.0021 s (2.4M iterations
first_value convert_to_state nulls=90%, filter=true
                        time:   [2.1172 µs 2.1355 µs 2.1555 µs]
change: [−0.6685% +0.5813% +1.7649%] (p = 0.37 > 0.05)
                        No change in performance detected.
Found 7 outliers among 100 measurements (7.00%)
  3 (3.00%) high mild
  4 (4.00%) high severe

Benchmarking first_value evaluate_bench nulls=90%, filter=true,
first(2): Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to
increase target time to 5.2s, enable flat sampling, or reduce sample
count to 60.
Benchmarking first_value evaluate_bench nulls=90%, filter=true,
first(2): Collecting 100 samples in estimated 5.2407 s (5050 it
first_value evaluate_bench nulls=90%, filter=true, first(2)
                        time:   [57.516 µs 58.757 µs 60.089 µs]
change: [−18.305% −15.732% −13.092%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  5 (5.00%) high mild

Benchmarking first_value evaluate_bench nulls=90%, filter=true, all:
Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to
increase target time to 5.3s, enable flat sampling, or reduce sample
count to 60.
Benchmarking first_value evaluate_bench nulls=90%, filter=true, all:
Collecting 100 samples in estimated 5.2574 s (5050 iterati
first_value evaluate_bench nulls=90%, filter=true, all
                        time:   [63.381 µs 65.433 µs 67.411 µs]
change: [+5.8137% +12.240% +19.268%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 10 outliers among 100 measurements (10.00%)
  1 (1.00%) high mild
  9 (9.00%) high severe

last_value convert_to_state nulls=0%, filter=false
                        time:   [100.59 µs 101.71 µs 103.06 µs]
change: [−0.8042% +0.1307% +1.0555%] (p = 0.79 > 0.05)
                        No change in performance detected.
Found 11 outliers among 100 measurements (11.00%)
  2 (2.00%) low mild
  6 (6.00%) high mild
  3 (3.00%) high severe

Benchmarking last_value evaluate_bench nulls=0%, filter=false, first(2):
Collecting 100 samples in estimated 7.3301 s (10k iter
last_value evaluate_bench nulls=0%, filter=false, first(2)
                        time:   [59.031 µs 60.067 µs 61.215 µs]
change: [−20.562% −15.398% −10.596%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking last_value evaluate_bench nulls=0%, filter=false, all:
Collecting 100 samples in estimated 7.4307 s (10k iteration
last_value evaluate_bench nulls=0%, filter=false, all
                        time:   [54.526 µs 55.568 µs 56.708 µs]
change: [−9.5126% −5.3244% −1.2387%] (p = 0.02 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  2 (2.00%) high mild
  1 (1.00%) high severe

last_value convert_to_state nulls=0%, filter=true
                        time:   [2.1428 µs 2.1838 µs 2.2289 µs]
change: [−5.7061% −3.1097% −0.6074%] (p = 0.02 < 0.05)
                        Change within noise threshold.

Benchmarking last_value evaluate_bench nulls=0%, filter=true, first(2):
Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to
increase target time to 5.1s, enable flat sampling, or reduce sample
count to 70.
Benchmarking last_value evaluate_bench nulls=0%, filter=true, first(2):
Collecting 100 samples in estimated 5.0694 s (5050 iter
last_value evaluate_bench nulls=0%, filter=true, first(2)
                        time:   [63.666 µs 66.076 µs 68.373 µs]
change: [−17.910% −13.091% −8.0825%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high mild

Benchmarking last_value evaluate_bench nulls=0%, filter=true, all:
Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to
increase target time to 5.1s, enable flat sampling, or reduce sample
count to 60.
Benchmarking last_value evaluate_bench nulls=0%, filter=true, all:
Collecting 100 samples in estimated 5.1148 s (5050 iteration
last_value evaluate_bench nulls=0%, filter=true, all
                        time:   [51.817 µs 52.699 µs 53.714 µs]
change: [−15.171% −12.649% −10.079%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  1 (1.00%) low mild
  3 (3.00%) high mild
  3 (3.00%) high severe

last_value convert_to_state nulls=90%, filter=false
                        time:   [107.51 µs 110.89 µs 114.78 µs]
change: [−0.7613% +1.8668% +4.7036%] (p = 0.17 > 0.05)
                        No change in performance detected.
Found 3 outliers among 100 measurements (3.00%)
  3 (3.00%) high mild

Benchmarking last_value evaluate_bench nulls=90%, filter=false,
first(2): Collecting 100 samples in estimated 9.6846 s (10k ite
last_value evaluate_bench nulls=90%, filter=false, first(2)
                        time:   [58.180 µs 58.969 µs 59.915 µs]
change: [−19.674% −16.125% −12.269%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

Benchmarking last_value evaluate_bench nulls=90%, filter=false, all:
Collecting 100 samples in estimated 9.2340 s (10k iteratio
last_value evaluate_bench nulls=90%, filter=false, all
                        time:   [52.432 µs 53.242 µs 54.103 µs]
change: [−12.306% −9.3583% −6.3687%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  7 (7.00%) high mild
  1 (1.00%) high severe

last_value convert_to_state nulls=90%, filter=true
                        time:   [2.0731 µs 2.0901 µs 2.1137 µs]
change: [−10.926% −9.9495% −8.9933%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  3 (3.00%) low mild
  2 (2.00%) high mild
  3 (3.00%) high severe

Benchmarking last_value evaluate_bench nulls=90%, filter=true, first(2):
Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to
increase target time to 5.7s, enable flat sampling, or reduce sample
count to 60.
Benchmarking last_value evaluate_bench nulls=90%, filter=true, first(2):
Collecting 100 samples in estimated 5.7484 s (5050 ite
last_value evaluate_bench nulls=90%, filter=true, first(2)
                        time:   [56.238 µs 57.148 µs 58.202 µs]
change: [−31.872% −29.982% −27.838%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  4 (4.00%) high mild
  3 (3.00%) high severe

Benchmarking last_value evaluate_bench nulls=90%, filter=true, all:
Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to
increase target time to 5.8s, enable flat sampling, or reduce sample
count to 60.
Benchmarking last_value evaluate_bench nulls=90%, filter=true, all:
Collecting 100 samples in estimated 5.7562 s (5050 iteratio
last_value evaluate_bench nulls=90%, filter=true, all
                        time:   [51.057 µs 52.223 µs 53.542 µs]
change: [−21.819% −19.372% −16.998%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  1 (1.00%) low mild
  4 (4.00%) high mild
  2 (2.00%) high severe

cargo bench --bench first_last -- --baseline main-first_last4 786.76s
user 16.16s system 135% cpu 9:50.80 total

</details>

## Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->

---------

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
@coderfender
feat : ABI upgrade from abi_stabby to stabby since abi_stable is no l…
af7904f 
…onger maintained (apache#21030)

## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->

- Closes apache#20863

## Rationale for this change

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->

## What changes are included in this PR?

  - Replace abi_stable types with stabby equivalents:
    - `RVec`  -> `stabby::vec::Vec`
    - `RString` -> `stabby::string::String`
- Add custom `FFI_Option` and `FFI_Result` types since we have raw
pointers and self referential pointers which can't implement `IStable`
trait needed for stabby
- Rename `macros: rresult` → `sresult`, `rresult_return `→
`sresult_return`
- Fix library path detection for debug/release builds in the example
loader
  - Update README 

<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

## Are these changes tested?

Yes . With df python bindings on DF 52 release 

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

## Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
@OlegWock
Add protobuf serialization/deserialization support for EmptyTable s…
f802ed1 
…cans (apache#20844)

## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->

I figured it will be easier to submit PR right away as change doesn't
look controversial. I'm happy to create an issue and link it here if
you'd prefer.

## Rationale for this change

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->

So short story is: in another project we'd like to use DataFusion's to
"build" operations on data and then submit resulting logical plan
_somewhere_ to execute (likely not using DF to actually execute the
query). Since those plans never meant to be executed by DF we use
`EmptyTable` as a base to bring schema to DF without any actual data.
`EmptyTable` scans not being serializable prevents us from sending those
plans to Python or over the wire. I believe this change makes
datafusion's LogicalPlan more portable and more usable outside of
datafusion's query executor.

Longer story: [VegaFusion](https://github.com/vega/vegafusion) does
server-side aggregation for Vega charts and is powered by DataFusion. We
recently added option to [use custom query/plan
executors](vega/vegafusion#573), which allows
user to pass a schema (without data) to VegaFusion which will add all
necessary aggregations (but not execute them) and return a logical plan
to user. They can then outsource this plan to custom query executor
(e.g. Spark). This is already implemented and works. However, since
VegaFusion is most commonly used through Python bindings, we'd like to
expose this API to Python too (and additionally as part of gPRC API too)
, which requires serializing built plans to protobuf. Currently we use
`EmptyTable` to bring schema without any data to DataFusion. But since
it can't be converted to protobuf, we're unable to expose this API. We
considered providing custom decoder/encoder, but that would work only
for gRPC case, but not Python as datafusion-python doesn't allow to
provide custom decoder as far as I understand.

## What changes are included in this PR?

<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

* Moved `EmptyTable` from `datafusion-core` into `datafusion-catalog`
and added backwards compatibility re-export (following pattern for other
table providers moved earlier)
* Added new `EmptyTableScanNode` to protobuf definitions
* Added encoding and decoding for new entity into `AsLogicalPlan for
LogicalPlanNode` implementation

## Are these changes tested?

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

I added two roundtrip tests for the new node

## Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

`EmptyTable` can be imported from `datafusion-catalog` crate now, but
old crate (`datafusion-core`) still re-exports it, so this shouldn't be
breaking change

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->

P.S. Just to be explicit, code itself was written mostly by LLM (as I'm
not that proficient in Rust yet). I did review and test it though
@kosiew
Support Dictionary Arrays in MIN/MAX Aggregates (apache#21315)
310dd5d 
## Which issue does this PR close?

* Closes apache#21150.

---

## Rationale for this change

The existing implementation of `min`/`max` does not correctly support
dictionary-encoded arrays. Previously, dictionary arrays were handled by
directly evaluating their underlying values array, which is semantically
incorrect because:

* It may include unreferenced values that do not appear in the logical
dataset
* It ignores nulls in the key array
* It does not preserve dictionary key semantics in scalar results

This leads to incorrect aggregation results for dictionary types.

This PR introduces a logical row-based evaluation for dictionary arrays
and ensures scalar comparisons correctly unwrap and rewrap dictionary
values when needed.

---

## What changes are included in this PR?

* Add logical row-based min/max computation (`scalar_row_extreme`) for:

  * Dictionary arrays
  * Struct, List, LargeList, and FixedSizeList types
* Replace previous dictionary handling that operated on `values()` with
correct row-wise evaluation
* Introduce `requires_logical_row_scan` to centralize fallback logic for
complex types
* Enhance scalar comparison logic:

  * Unwrap dictionary scalars before comparison
* Rewrap results when both inputs are dictionaries with matching key
types
  * Validate key type compatibility
* Improve error messaging for incompatible scalar comparisons
* Remove obsolete `min_max_batch_generic` implementation

---

## Are these changes tested?

Yes. Comprehensive tests have been added to validate correctness across
multiple scenarios:

* Basic dictionary min/max behavior
* Handling of null keys and null values
* Ignoring unreferenced dictionary values
* Multi-batch aggregation behavior
* Float dictionary handling including `NaN` and infinities

These tests ensure correctness and guard against regressions.

---

## Are there any user-facing changes?

Yes. The behavior of `min` and `max` on dictionary-encoded arrays is
now:

* Correct and semantically aligned with logical row values
* Consistent with other data types

Previously incorrect results may now differ, which is a correctness fix
rather than a breaking API change.

---

## LLM-generated code disclosure

This PR includes LLM-generated code and comments. All LLM-generated
content has been manually reviewed and tested.
@Jefffrey @alamb
Fix some GH action permission issues identified by CodeQL (apache#21838)
6a09260 
## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->

N/A

## Rationale for this change

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->

<img width="302" height="45" alt="image"
src="https://github.com/user-attachments/assets/fed0ded5-5ef5-4e10-9bd6-bb3391470ee7"
/>

Lets work on getting this number down to 0

## What changes are included in this PR?

<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

Fix some permissions to be minimum required in GH action workflows

## Are these changes tested?

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

All affected workflows were run on this commit:
apache@48ffe3f

- Since some workflows trigger on certain changes only

- large files successful:
https://github.com/apache/datafusion/actions/runs/24923629680/job/72989644694?pr=21838
- docs pr successful:
https://github.com/apache/datafusion/actions/runs/24923629679/job/72989644704?pr=21838
- dependencies:
- circular check successful:
https://github.com/apache/datafusion/actions/runs/24923629662/job/72989644657?pr=21838
- unused check successful:
https://github.com/apache/datafusion/actions/runs/24923629662/job/72989644656?pr=21838
- audit successful:
https://github.com/apache/datafusion/actions/runs/24923629664/job/72989644667?pr=21838
- dev:
- license check successful:
https://github.com/apache/datafusion/actions/runs/24923629678/job/72989644714?pr=21838
- prettier successful:
https://github.com/apache/datafusion/actions/runs/24923629678/job/72989644708?pr=21838
- asf yaml check successful:
https://github.com/apache/datafusion/actions/runs/24923629678/job/72989644706?pr=21838
- typos successful:
https://github.com/apache/datafusion/actions/runs/24923629678/job/72989644701?pr=21838

## Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->

---------

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
@Scolliq @alamb
perf(spark): use 256-entry byte-pair table in hex encoding (apache#21836
54a5515 
)

Refs apache#15986.

**Why:** `spark_hex` walked one nibble at a time — two `HEX_CHARS[i]`
lookups and two `Vec::push` calls per input byte. The hot loop flattens
into one indexed load and one `extend_from_slice` per byte with a
precomputed table.

**What changed:** added `HEX_LOOKUP_LOWER` / `HEX_LOOKUP_UPPER` as
`[[u8; 2]; 256]` const tables built at compile time. Bytes path now does
a single lookup + 2-byte extend per input byte. The int64 path consumes
two nibbles per iteration via the same table, with a fall-through for
the high nibble. Behaviour for `0`, `i64::MAX`, `i64::MIN`, `-1`
preserved.

**Tests:** extended `test_hex_int64` to cover edge values; new
`test_hex_lookup_table_covers_all_bytes` cross-checks every entry
against `format!("{:02X/x}")`; new
`test_spark_hex_binary_round_trip_all_bytes` feeds all 256 byte values
through `spark_hex` and verifies the result.

`cargo test -p datafusion-spark --lib hex` → 8 pass. `cargo clippy
--all-features --all-targets` clean. `cargo bench --no-run` builds —
existing `benches/hex.rs` already covers
Int64/Utf8/Utf8View/LargeUtf8/Binary/LargeBinary plus dict paths.

**Not in this PR:** the apache#15947 review also flagged Utf8View output and
dictionary-key reuse — those felt worth their own PRs to keep this
focused on the per-byte hot path.

---------

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
@gstvg @rluvaton
@martin-g @LiaCastaneda
Add lambda support and array_transform udf (apache#21679)
bbf67d9 
This a clean version of apache#18921 to make it easier to review

**this is a breaking change due to adding variant to `Expr` enum, new
methods on traits `Session`, `FunctionRegistry` and `ContextProvider`
and a new arg on `TaskContext::new`**

This PR adds support for lambdas and the `array_transform` function used
to test the lambda implementation.

Example usage:

```sql
SELECT array_transform([2, 3], v -> v != 2);

[false, true]

-- arbitrally nested lambdas are also supported
SELECT array_transform([[[2, 3]]], m -> array_transform(m, l -> array_transform(l, v -> v*2)));

[[[4, 6]]]
```
Note: column capture has been removed for now and will be added on a
follow on PR, see apache#21172

Some comments on code snippets of this doc show what value each struct,
variant or field would hold after planning the first example above. Some
literals are simplified pseudo code

3 new `Expr` variants are added, `HigherOrderFunction`, owing a new
trait `HigherOrderUDF`, which is like a `ScalarFunction`/`ScalarUDFImpl`
with support for lambdas, `Lambda`, for the lambda body and it's
parameters names, and `LambdaVariable`, which is like `Column` but for
lambdas parameters.

Their logical representations:

```rust
enum Expr {
    // array_transform([2, 3], v -> v != 2)
    HigherOrderFunction(HigherOrderFunction),
    // v -> v != 2
    Lambda(Lambda),
    // v, of the lambda body: v != 2
    LambdaVariable(LambdaVariable),
   ...
}

// array_transform([2, 3], v -> v != 2)
struct HigherOrderFunction {
    // global instance of array_transform
    pub func: Arc<dyn HigherOrderUDF>,
    // [Expr::ScalarValue([2, 3]), Expr::Lambda(v -> v != 2)]
    pub args: Vec<Expr>,
}

// v -> v != 2
struct Lambda {
    // ["v"]
    pub params: Vec<String>,
    // v != 2
    pub body: Box<Expr>,
}

// v, of the lambda body: v != 2
struct LambdaVariable {
    // "v"
    pub name: String,
    // Field::new("", DataType::Int32, false) 
    // Note: a follow on PR will make this field optional
    // to free expr_api from specifying it beforehand, 
    // and add resolve_lambda_variables method to Expr,
    // similar to Expr::Placeholder, see apache#21172
    pub field: FieldRef, 
    pub spans: Spans,
}

```

The example would be planned into a tree like this:

```
HigherOrderFunctionExpression
  name: array_transform
  children:
    1. ListExpression [2,3]
    2. LambdaExpression
         parameters: ["v"]
         body:
            BinaryExpression (!=)
              left:
                 LambdaVariableExpression("v", Field::new("", Int32, false))
              right:
                 LiteralExpression("2")
```

The physical counterparts definition:

```rust

struct HigherOrderFunctionExpr {
    // global instance of array_transform
    fun: Arc<dyn HigherOrderUDF>,
    // "array_transform"
    name: String,
    // [LiteralExpr([2, 3], LambdaExpr("v -> v != 2"))]
    args: Vec<Arc<dyn PhysicalExpr>>,
    // [1], the positions at args that contains lambdas
    lambda_positions: Vec<usize>,
    // Field::new("", DataType::new_list(DataType::Boolean, false), false)
    return_field: FieldRef,
    config_options: Arc<ConfigOptions>, 
}


struct LambdaExpr {
    // ["v"]
    params: Vec<String>,
    // v -> v != 2
    body: Arc<dyn PhysicalExpr>,
}

struct LambdaVariable {
    // Field::new("v", DataType::Int32, false)
    field: FieldRef,
    // 0, the first and only parameter, "v"
    index: usize,
}
```

Note: For those who primarly wants to check if this lambda
implementation supports their usecase and don't want to spend much time
here, it's okay to skip most collapsed blocks, as those serve mostly to
help code reviewers, with the exception of `HigherOrderUDF` and the
`array_transform` implementation of `HigherOrderUDF` relevant methods,
collapsed due to their size


The added `HigherOrderUDF` trait is almost a clone of `ScalarUDFImpl`,
with the exception of:
1. `return_field_from_args` and `invoke_with_args`, where now
`args.args` is a list of enums with two variants: `Value` or `Lambda`
instead of a list of values
2. the addition of `lambda_parameters`, which return a `Field` for each
parameter supported for every lambda argument based on the `Field` of
the non lambda arguments
3. the removal of `return_field` and the deprecated ones `is_nullable`
and `display_name`.
4. Not yet includes analogues to the methods preimage, placement,
evaluate_bounds, propagate_constraints, output_ordering and
preserves_lex_ordering

<details><summary>HigherOrderUDF</summary>

```rust

trait HigherOrderUDF {
    /// Return the field of all the parameters supported by all the supported lambdas of this function
    /// based on the field of the value arguments. If a lambda support multiple parameters, or if multiple
    /// lambdas are supported and some are optional, all should be returned,
    /// regardless of whether they are used on a particular invocation
    ///
    /// Tip: If you have a [`HigherOrderFunction`] invocation, you can call the helper
    /// [`HigherOrderFunction::lambda_parameters`] instead of this method directly
    ///
    /// [`HigherOrderFunction`]: crate::expr::HigherOrderFunction
    /// [`HigherOrderFunction::lambda_parameters`]: crate::expr::HigherOrderFunction::lambda_parameters
    ///
    /// Example for array_transform:
    ///
    /// `array_transform([2.0, 8.0], v -> v > 4.0)`
    ///
    /// ```ignore
    /// let lambda_parameters = array_transform.lambda_parameters(&[
    ///      Arc::new(Field::new("", DataType::new_list(DataType::Float32, false))), // the Field of the literal `[2, 8]`
    /// ])?;
    ///
    /// assert_eq!(
    ///      lambda_parameters,
    ///      vec![
    ///         // the lambda supported parameters, regardless of how many are actually used
    ///         vec![
    ///             // the value being transformed
    ///             Field::new("", DataType::Float32, false),
    ///         ]
    ///      ]
    /// )
    /// ```
    ///
    /// The implementation can assume that some other part of the code has coerced
    /// the actual argument types to match [`Self::signature`].
    fn lambda_parameters(&self, value_fields: &[FieldRef]) -> Result<Vec<Vec<Field>>>;
    fn return_field_from_args(&self, args: LambdaReturnFieldArgs) -> Result<FieldRef>;
    fn invoke_with_args(&self, args: HigherOrderFunctionArgs) -> Result<ColumnarValue>;
   // ... omitted methods that are similar in ScalarUDFImpl
}

/// An argument to a lambda function
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum ValueOrLambda<V, L> {
    /// A value with associated data
    Value(V),
    /// A lambda with associated data
    Lambda(L),
}

/// Information about arguments passed to the function
///
/// This structure contains metadata about how the function was called
/// such as the type of the arguments, any scalar arguments and if the
/// arguments can (ever) be null
///
/// See [`HigherOrderUDF::return_field_from_args`] for more information
#[derive(Clone, Debug)]
pub struct LambdaReturnFieldArgs<'a> {
    /// The data types of the arguments to the function
    ///
    /// If argument `i` to the function is a lambda, it will be the field of the result of the
    /// lambda if evaluated with the parameters returned from [`HigherOrderUDF::lambda_parameters`]
    ///
    /// For example, with `array_transform([1], v -> v == 5)`
    /// this field will be `[
    ///     ValueOrLambda::Value(Field::new("", DataType::List(DataType::Int32), false)),
    ///     ValueOrLambda::Lambda(Field::new("", DataType::Boolean, false))
    /// ]`
    pub arg_fields: &'a [ValueOrLambda<FieldRef, FieldRef>],
    /// Is argument `i` to the function a scalar (constant)?
    ///
    /// If the argument `i` is not a scalar, it will be None
    ///
    /// For example, if a function is called like `array_transform([1], v -> v == 5)`
    /// this field will be `[Some(ScalarValue::List(...), None]`
    pub scalar_arguments: &'a [Option<&'a ScalarValue>],
}

/// Arguments passed to [`HigherOrderUDF::invoke_with_args`] when invoking a
/// lambda function.
#[derive(Debug, Clone)]
pub struct HigherOrderFunctionArgs {
    /// The evaluated arguments and lambdas to the function
    pub args: Vec<ValueOrLambda<ColumnarValue, LambdaArgument>>,
    /// Field associated with each arg, if it exists
    /// For lambdas, it will be the field of the result of
    /// the lambda if evaluated with the parameters
    /// returned from [`HigherOrderUDF::lambda_parameters`]
    pub arg_fields: Vec<ValueOrLambda<FieldRef, FieldRef>>,
    /// The number of rows in record batch being evaluated
    pub number_rows: usize,
    /// The return field of the lambda function returned
    /// (from `return_field_from_args`) when creating the
    /// physical expression from the logical expression
    pub return_field: FieldRef,
    /// The config options at execution time
    pub config_options: Arc<ConfigOptions>,
}

/// A lambda argument to a HigherOrderFunction
#[derive(Clone, Debug)]
pub struct LambdaArgument {
    /// The parameters defined in this lambda
    ///
    /// For example, for `array_transform([2], v -> -v)`,
    /// this will be `vec![Field::new("v", DataType::Int32, true)]`
    params: Vec<FieldRef>,
    /// The body of the lambda
    ///
    /// For example, for `array_transform([2], v -> -v)`,
    /// this will be the physical expression of `-v`
    body: Arc<dyn PhysicalExpr>,
}

impl LambdaArgument {
    /// Evaluate this lambda
    /// `args` should evalute to the value of each parameter
    /// of the correspondent lambda returned in [HigherOrderUDF::lambda_parameters].
    pub fn evaluate(
        &self,
        args: &[&dyn Fn() -> Result<ArrayRef>],
    ) -> Result<ColumnarValue> {
        let columns = args
            .iter()
            .take(self.params.len())
            .map(|arg| arg())
            .collect::<Result<_>>()?;

        let schema = Arc::new(Schema::new(self.params.clone()));

        let batch = RecordBatch::try_new(schema, columns)?;

        self.body.evaluate(&batch)
    }
}
```


</details>

<details><summary>array_transform lambda_parameters
implementation</summary>

```rust
impl HigherOrderUDF for ArrayTransform {
fn lambda_parameters(&self, value_fields: &[FieldRef]) -> Result<Vec<Vec<Field>>> {
        let list = if value_fields.len() == 1 {
            &value_fields[0]
        } else {
            return plan_err!(
                "{} function requires 1 value arguments, got {}",
                self.name(),
                value_fields.len()
            );
        };

        let field = match list.data_type() {
            DataType::List(field) => field,
            DataType::LargeList(field) => field,
            DataType::FixedSizeList(field, _) => field,
            _ => return plan_err!("expected list, got {list}"),
        };

        // we don't need to check whether the lambda contains more than two parameters,
        // e.g. array_transform([], (v, i, j) -> v+i+j), as datafusion will do that for us
        let value = Field::new("", field.data_type().clone(), field.is_nullable())
            .with_metadata(field.metadata().clone());

        Ok(vec![vec![value]])
    }
}
```

</details>

<details><summary>array_transform return_field_from_args
implementation</summary>

```rust
fn value_lambda_pair<'a, V: Debug, L: Debug>(
    name: &str,
    args: &'a [ValueOrLambda<V, L>],
) -> Result<(&'a V, &'a L)> {
    let [value, lambda] = take_function_args(name, args)?;

    let (ValueOrLambda::Value(value), ValueOrLambda::Lambda(lambda)) = (value, lambda)
    else {
        return plan_err!(
            "{name} expects a value followed by a lambda, got {value:?} and {lambda:?}"
        );
    };

    Ok((value, lambda))
}

impl HigherOrderUDF for ArrayTransform {
    fn return_field_from_args(
        &self,
        args: HigherOrderReturnFieldArgs,
    ) -> Result<Arc<Field>> {
        let (list, lambda) = value_lambda_pair(self.name(), args.arg_fields)?;

        // lambda is the resulting field of executing the lambda body
        // with the parameters returned in lambda_parameters
        let field = Arc::new(Field::new(
            Field::LIST_FIELD_DEFAULT_NAME,
            lambda.data_type().clone(),
            lambda.is_nullable(),
        ));

        let return_type = match list.data_type() {
            DataType::List(_) => DataType::List(field),
            DataType::LargeList(_) => DataType::LargeList(field),
            DataType::FixedSizeList(_, size) => DataType::FixedSizeList(field, *size),
            other => plan_err!("expected list, got {other}")?,
        };

        Ok(Arc::new(Field::new("", return_type, list.is_nullable())))
    }
}
```

</details>

<details><summary>array_transform invoke_with_args
implementation</summary>


```rust
impl HigherOrderUDF for ArrayTransform {
fn invoke_with_args(&self, args: HigherOrderFunctionArgs) -> Result<ColumnarValue> {
        let (list, lambda) = value_lambda_pair(self.name(), &args.args)?;

        let list_array = list.to_array(args.number_rows)?;

        // Fast path for fully null input array and also the only way to safely work with
        // a fully null fixed size list array as it can't be handled by remove_list_null_values below
        if list_array.null_count() == list_array.len() {
            return Ok(ColumnarValue::Array(new_null_array(
                args.return_type(),
                list_array.len(),
            )));
        }

        // as per list_values docs, if list_array is sliced, list_values will be sliced too,
        // so before constructing the transformed array below, we must adjust the list offsets with
        // adjust_offsets_for_slice
        let list_values = list_values(&list_array)?;

        // by passing closures, lambda.evaluate can evaluate only those actually needed
        let values_param = || Ok(Arc::clone(&list_values));

        // call the transforming lambda
        let transformed_values = lambda
            .evaluate(&[&values_param])?
            .into_array(list_values.len())?;

        let field = match args.return_field.data_type() {
            DataType::List(field)
            | DataType::LargeList(field)
            | DataType::FixedSizeList(field, _) => Arc::clone(field),
            _ => {
                return exec_err!(
                    "{} expected ScalarFunctionArgs.return_field to be a list, got {}",
                    self.name(),
                    args.return_field
                );
            }
        };

        let transformed_list = match list_array.data_type() {
            DataType::List(_) => {
                let list = list_array.as_list();

                // since we called list_values above which would return sliced values for
                // a sliced list, we must adjust the offsets here as otherwise they would be invalid
                let adjusted_offsets = adjust_offsets_for_slice(list);

                Arc::new(ListArray::new(
                    field,
                    adjusted_offsets,
                    transformed_values,
                    list.nulls().cloned(),
                )) as ArrayRef
            }
            DataType::LargeList(_) => {
                let large_list = list_array.as_list();

                // since we called list_values above which would return sliced values for
                // a sliced list, we must adjust the offsets here as otherwise they would be invalid
                let adjusted_offsets = adjust_offsets_for_slice(large_list);

                Arc::new(LargeListArray::new(
                    field,
                    adjusted_offsets,
                    transformed_values,
                    large_list.nulls().cloned(),
                ))
            }
            DataType::FixedSizeList(_, value_length) => {
                Arc::new(FixedSizeListArray::new(
                    field,
                    *value_length,
                    transformed_values,
                    list_array.as_fixed_size_list().nulls().cloned(),
                ))
            }
            other => exec_err!("expected list, got {other}")?,
        };

        Ok(ColumnarValue::Array(transformed_list))
    }
}
```

</details>

<details><summary>How relevant HigherOrderUDF methods would be called
and what they would return during planning and evaluation of the
example</summary>


```rust
// this is called at sql planning
let lambda_parameters = lambda_udf.lambda_parameters(&[
    Field::new("", DataType::new_list(DataType::Int32, false), false), // the Field of the [2, 3] literal
])?;

assert_eq!(
    lambda_parameters,
    vec![
            // the parameters that *can* be declared on the lambda, and not only 
            // those actually declared: the implementation doesn't need to care 
            // about it
            vec![
                Field::new("", DataType::Int32, false), // the list inner value
            ]]
);



// this is called every time ExprSchemable is called on a HigherOrderFunction
let return_field = array_transform.return_field_from_args(&LambdaReturnFieldArgs {
    arg_fields: &[
        ValueOrLambda::Value(Field::new("", DataType::new_list(DataType::Int32, false), false)),
        ValueOrLambda::Lambda(Field::new("", DataType::Boolean, false)), // the return_field of the expression "v != 2" when "v" is of the type returned in lambda_parameters
    ],
    scalar_arguments // irrelevant
})?;

assert_eq!(return_field, Field::new("", DataType::new_list(DataType::Boolean, false), false));



let value = array_transform.evaluate(&HigherOrderFunctionArgs {
    args: vec![
        ValueOrLambda::Value(List([2, 3])),
        ValueOrLambda::Lambda(LambdaArgument of `v -> v != 2`),
    ],
    arg_fields, // same as above
    number_rows: 1,
    return_field, // same as above
    config_options, // irrelevant
})?;

assert_eq!(value, BooleanArray::from([false, true]))
```

</details>
<br>
<br>

A pair HigherOrderUDF/HigherOrderUDFImpl like ScalarFunction was not
used because those exist only [to maintain backwards compatibility with
the older
API](https://docs.rs/datafusion/latest/datafusion/logical_expr/struct.ScalarUDF.html#api-note)
apache#8045

</details>
<br>

Why `LambdaVariable` and not `Column`:

Existing tree traversals that operate on columns would break if some
column nodes referenced to a lambda parameter and not a real column. In
the example query, projection pushdown would try to push the lambda
parameter "v", which won't exist in table "t".

Example of code of another traversal that would break:

```rust
fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter {
    let mut used_columns = HashSet::new();
    expr.apply(|expr| {
        if let Some(col) = expr.as_any().downcast_ref::<Column>() {
            // if this is a lambda column, this function will break
            used_columns.insert(col.index());
        }
        Ok(TreeNodeRecursion::Continue)
    });
    ...
}
```

Furthermore, the implemention of `ExprSchemable` and
`PhysicalExpr::return_field` for `Column` expects that the schema it
receives as a argument contains an entry for its name, which is not the
case for lambda parameters.

By including a `FieldRef` on `LambdaVariable` that should be resolved
during construction time in the sql planner, `ExprSchemable` and
`PhysicalExpr::return_field` simply return it's own Field:

<details><summary>LambdaVariable ExprSchemable and
PhysicalExpr::return_field implementation </summary>

```rust
impl ExprSchemable for Expr {
   fn to_field(
        &self,
        schema: &dyn ExprSchema,
    ) -> Result<(Option<TableReference>, Arc<Field>)> {
        let (relation, schema_name) = self.qualified_name();
        let field = match self {
           Expr::LambdaVariable(l) => Ok(Arc::clone(&l.field)),
           ...
        }?;

        Ok((
            relation,
            Arc::new(field.as_ref().clone().with_name(schema_name)),
        ))
    }
    ...
}

impl PhysicalExpr for LambdaVariable {
    fn return_field(&self, _input_schema: &Schema) -> Result<FieldRef> {
        Ok(Arc::clone(&self.field))
    }
    ...
}
```

</details>
<br>

<details><summary>Possible alternatives discarded due to complexity,
requiring downstream changes and implementation size:</summary>

1. Add a new set of TreeNode methods that provides the set of lambdas
parameters names seen during the traversal, so column nodes can be
tested if they refer to a regular column or to a lambda parameter. Any
downstream user that wants to support lambdas would need use those
methods instead of the existing ones. This also would add 1k+ lines to
the PR.

```rust
impl Expr {
    pub fn transform_with_lambdas_params<
        F: FnMut(Self, &HashSet<String>) -> Result<Transformed<Self>>,
    >(
        self,
        mut f: F,
    ) -> Result<Transformed<Self>> {}
}
```

How minimize_join_filter would looks like:


```rust
fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter {
    let mut used_columns = HashSet::new();
    expr.apply_with_lambdas_params(|expr, lambdas_params| {
        if let Some(col) = expr.as_any().downcast_ref::<Column>() {
            // dont include lambdas parameters
            if !lambdas_params.contains(col.name()) {
                used_columns.insert(col.index());
            }
        }
        Ok(TreeNodeRecursion::Continue)
    })
    ...
}
```

2. Add a flag to the Column node indicating if it refers to a lambda
parameter. Still requires checking for it on existing tree traversals
that works on Columns (30+) and also downstream.

```rust
//logical
struct Column {
    pub relation: Option<TableReference>,
    pub name: String,
    pub spans: Spans,
    pub is_lambda_parameter: bool,
}

//physical
struct Column {
    name: String,
    index: usize,
    is_lambda_parameter: bool,
}
```


How minimize_join_filter would look like:

```rust
fn minimize_join_filter(expr: Arc<dyn PhysicalExpr>, ...) -> JoinFilter {
    let mut used_columns = HashSet::new();
    expr.apply(|expr| {
        if let Some(col) = expr.as_any().downcast_ref::<Column>() {
            // dont include lambdas parameters
            if !col.is_lambda_parameter {
                used_columns.insert(col.index());
            }
        }
        Ok(TreeNodeRecursion::Continue)
    })
    ...
}
```


1. Add a new set of TreeNode methods that provides a schema that
includes the lambdas parameters for the scope of the node being
visited/transformed:

```rust
impl Expr {
    pub fn transform_with_schema<
        F: FnMut(Self, &DFSchema) -> Result<Transformed<Self>>,
    >(
        self,
        schema: &DFSchema,
        f: F,
    ) -> Result<Transformed<Self>> { ... }
    ... other methods
}
```

For any given HigherOrderFunction found during the traversal, a new
schema is created for each lambda argument that contains it's parameter,
returned from HigherOrderUDF::lambda_parameters
How it would look like:

```rust

pub fn infer_placeholder_types(self, schema: &DFSchema) -> Result<(Expr, bool)> {
        let mut has_placeholder = false;
        // Provide the schema as the first argument. 
        // Transforming closure receive an adjusted_schema as argument
        self.transform_with_schema(schema, |mut expr, adjusted_schema| {
            match &mut expr {
                // Default to assuming the arguments are the same type
                Expr::BinaryExpr(BinaryExpr { left, op: _, right }) => {
                    // use adjusted_schema and not schema. Those expressions may contain 
                    // columns referring to a lambda parameter, which Field would only be
                    // available in adjusted_schema and not in schema
                    rewrite_placeholder(left.as_mut(), right.as_ref(), adjusted_schema)?;
                    rewrite_placeholder(right.as_mut(), left.as_ref(), adjusted_schema)?;
                }
    ....

```

2. Make available trought LogicalPlan and ExecutionPlan nodes a schema
that includes all lambdas parameters from all expressions owned by the
node, and use this schema for tree traversals. For nodes which won't own
any expression, the regular schema can be returned


```rust
impl LogicalPlan {
    fn lambda_extended_schema(&self) -> &DFSchema;
}

trait ExecutionPlan {
    fn lambda_extended_schema(&self) -> &DFSchema;
}

//usage
impl LogicalPlan {
    pub fn replace_params_with_values(
            self,
            param_values: &ParamValues,
        ) -> Result<LogicalPlan> {
            self.transform_up_with_subqueries(|plan| {
                // use plan.lambda_extended_schema() containing lambdas parameters
                // instead of plan.schema() which wont
                let lambda_extended_schema = Arc::clone(plan.lambda_extended_schema());
                let name_preserver = NamePreserver::new(&plan);
                plan.map_expressions(|e| {
                    // if this expression is child of lambda and contain columns referring it's parameters
                    // the lambda_extended_schema already contain them
                    let (e, has_placeholder) = e.infer_placeholder_types(&lambda_extended_schema)?;
    ....

```
</details>
<br>

---------

Co-authored-by: Raz Luvaton <16746759+rluvaton@users.noreply.github.com>
Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
Co-authored-by: Lía Adriana <lia.castaneda@datadoghq.com>
@sgrebnov
feat(unparser): Keep inner join Filter → TableScan predicates to `W…
c2c0773 
…HERE` instead of moving to `JOIN ON` (apache#21694)

## Which issue does this PR close?

Partially addresses apache#13156
(inner joins only; outer joins require additional work)

When the DataFusion optimizer pushes filter predicates into `TableScan`
nodes (e.g. via `FilterPushdown`), the unparser's
`try_transform_to_simple_table_scan_with_filters` extracts those filters
and then always folds them into the `JOIN ON` clause. This is
problematic when the extracted filters contain subquery expressions
(scalar subqueries, `IN`, `EXISTS`), because some SQL backends — notably
BigQuery — reject subqueries inside `JOIN ON` predicates.

This currently breaks 5 TPC-H queries (Q2, Q16, Q17, Q18, Q21) when
unparsed SQL is sent to BigQuery.

We did attempt to fix this
(apache#13496) by moving **all**
filters to `WHERE`, which broke `LEFT`/`RIGHT`/`FULL` join semantics
(moving a filter from `ON` to `WHERE` changes the result for outer
joins, as demonstrated in
apache#13132).

## What changes are included in this PR?

For **inner joins only**, `table_scan_filters` extracted by
`try_transform_to_simple_table_scan_with_filters` are now placed in the
`WHERE` clause instead of the `JOIN ON` clause. This is safe because
`ON` and `WHERE` are semantically equivalent for inner joins.

For non-inner joins (`LEFT`, `RIGHT`, `FULL`), the existing behavior is
preserved — filters remain in `JOIN ON` — since moving them to `WHERE`
would change query semantics.

## Are these changes tested?

Yes.

- Added a test case in `test_join_with_table_scan_filters` that
constructs an inner join where the right side has a
`table_scan_with_filters` containing a scalar subquery. Verifies the
subquery predicate appears in `WHERE`, not `JOIN ON`.
- Updated existing snapshots in `test_join_with_table_scan_filters`
reflecting that `table_scan_filters` now appear in `WHERE` for inner
joins.

## Are there any user-facing changes?

SQL generated by the unparser for inner joins may now place `TableScan`
pushdown filters in the `WHERE` clause instead of the `JOIN ON` clause
(similar to changes in `test_join_with_table_scan_filters`)

## Alternatives considered

An alternative approach considered is to introduce a
`supports_subquery_in_join_predicate` dialect flag that only moves
subquery-containing filters to `WHERE` when the dialect opts in (e.g.
`BigQueryDialect`), preserving existing behavior for all other dialects.

Example implementation: spiceai#151

Current approach was chosen due to 
- simplicity: no new dialect flag, fewer code paths, simple change.
- performance (potentially): as noted in
[apache#13156](apache#13156), placing
filters in `WHERE` can trigger filter pushdown on the target backend,
which is a potential performance win.
@tabac @gabotechs
Add support for nested types to nullif. (apache#21764)
22bb4e6 
## Which issue does this PR close?

- Closes apache#21763

## Rationale for this change

Add support for nested types to the `nullif` UDF.

## Are these changes tested?

Unit tests included.

## Are there any user-facing changes?

No changes to the function's signature.

---------

Co-authored-by: Gabriel <45515538+gabotechs@users.noreply.github.com>
@neilconway @martin-g
perf: Optimize substr_index to use bulk-NULL string builder (apache…
ec92925 
…#21877)

## Which issue does this PR close?

- Closes apache#21876.

## Rationale for this change

As with other recent optimizations, we can optimize NULL handling in
`substr_index` by using the new bulk-NULL string builders.

Benchmarks:

Utf8
  - utf8_100_array_long_delimiter: 10.0 µs → 10.1 µs (+1.00%)
  - utf8_100_array_single_delimiter: 2.9 µs → 2.5 µs (−13.79%)
  - utf8_100_scalar_long_delimiter_neg: 4.1 µs → 3.5 µs (−14.63%)
  - utf8_100_scalar_long_delimiter_pos: 2.9 µs → 2.7 µs (−6.90%)
  - utf8_100_scalar_single_delimiter_neg: 2.2 µs → 1.993 µs (−9.41%)
  - utf8_100_scalar_single_delimiter_pos: 2.1 µs → 1.845 µs (−12.13%)
  - utf8_1000_array_long_delimiter: 101.0 µs → 101.1 µs (+0.10%)
  - utf8_1000_array_single_delimiter: 36.8 µs → 31.7 µs (−13.86%)
  - utf8_1000_scalar_long_delimiter_neg: 38.9 µs → 36.9 µs (−5.14%)
  - utf8_1000_scalar_long_delimiter_pos: 25.1 µs → 23.3 µs (−7.17%)
  - utf8_1000_scalar_single_delimiter_neg: 19.3 µs → 17.7 µs (−8.29%)
  - utf8_1000_scalar_single_delimiter_pos: 18.2 µs → 16.6 µs (−8.79%)
  - utf8_10000_array_long_delimiter: 1083.4 µs → 1038.2 µs (−4.17%)
  - utf8_10000_array_single_delimiter: 461.8 µs → 414.7 µs (−10.20%)
  - utf8_10000_scalar_long_delimiter_neg: 392.4 µs → 379.3 µs (−3.34%)
  - utf8_10000_scalar_long_delimiter_pos: 246.5 µs → 227.4 µs (−7.75%)
  - utf8_10000_scalar_single_delimiter_neg: 191.3 µs → 177.5 µs (−7.21%)
  - utf8_10000_scalar_single_delimiter_pos: 179.4 µs → 168.8 µs (−5.91%)

Utf8View
  - utf8view_100_array_long_delimiter: 9.5 µs → 9.8 µs (+3.16%)
  - utf8view_100_array_single_delimiter: 2.6 µs → 2.6 µs (0.00%)
  - utf8view_100_scalar_long_delimiter_neg: 4.0 µs → 4.0 µs (0.00%)
  - utf8view_100_scalar_long_delimiter_pos: 2.8 µs → 2.8 µs (0.00%)
  - utf8view_100_scalar_single_delimiter_neg: 2.3 µs → 2.3 µs (0.00%)
  - utf8view_100_scalar_single_delimiter_pos: 2.2 µs → 2.1 µs (−4.55%)
  - utf8view_1000_array_long_delimiter: 94.8 µs → 99.2 µs (+4.64%)
  - utf8view_1000_array_single_delimiter: 31.5 µs → 32.0 µs (+1.59%)
  - utf8view_1000_scalar_long_delimiter_neg: 38.7 µs → 39.0 µs (+0.78%)
  - utf8view_1000_scalar_long_delimiter_pos: 25.4 µs → 25.4 µs (0.00%)
- utf8view_1000_scalar_single_delimiter_neg: 21.4 µs → 21.8 µs (+1.87%)
- utf8view_1000_scalar_single_delimiter_pos: 20.8 µs → 20.9 µs (+0.48%)
  - utf8view_10000_array_long_delimiter: 998.4 µs → 1025.4 µs (+2.70%)
  - utf8view_10000_array_single_delimiter: 414.9 µs → 415.7 µs (+0.19%)
- utf8view_10000_scalar_long_delimiter_neg: 393.7 µs → 395.9 µs (+0.56%)
- utf8view_10000_scalar_long_delimiter_pos: 253.4 µs → 252.7 µs (−0.28%)
- utf8view_10000_scalar_single_delimiter_neg: 214.5 µs → 217.3 µs
(+1.31%)
- utf8view_10000_scalar_single_delimiter_pos: 207.9 µs → 208.7 µs
(+0.38%)

This PR doesn't touch the Utf8View code path, so the Utf8View
regressions above are likely measurement noise.

## What changes are included in this PR?

* Optimize `substr_index` by switching from Arrow string builders to
bulk-NULL string builders

## Are these changes tested?

Yes, covered by existing tests.

## Are there any user-facing changes?

No.

---------

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
@alamb
Update documentation for PhysicalExpr::evaluate_bounds (apache#21879)
686d617 
## Which issue does this PR close?

- Related to apache#21651 from
@Dandandan

## Rationale for this change

While reviewing apache#21651 I found
that `sin(x)` is hard coded to always return `[-1, 1]` as its bounds
regardless of its input:


https://github.com/apache/datafusion/blob/5901df58b21b8b4e36011744e7ddc17bcb6a37b3/datafusion/functions/src/math/bounds.rs#L27-L32

This is not clear from the docs and
apache#21651 was assuming something
different

## What changes are included in this PR?

1. Update the documentation to reflect the current state of the code
(the bound are not exact)

## Are these changes tested?

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

## Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
@dependabot
chore(deps): bump taiki-e/install-action from 2.75.18 to 2.75.23 (apa…
2a2b5b0 
…che#21887)

Bumps
[taiki-e/install-action](https://github.com/taiki-e/install-action) from
2.75.18 to 2.75.23.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/taiki-e/install-action/releases">taiki-e/install-action's
releases</a>.</em></p>
<blockquote>
<h2>2.75.23</h2>
<ul>
<li>
<p>Update <code>vacuum@latest</code> to 0.26.2.</p>
</li>
<li>
<p>Update <code>tombi@latest</code> to 0.9.24.</p>
</li>
<li>
<p>Update <code>mise@latest</code> to 2026.4.22.</p>
</li>
<li>
<p>Update <code>martin@latest</code> to 1.7.0.</p>
</li>
<li>
<p>Update <code>git-cliff@latest</code> to 2.13.1.</p>
</li>
<li>
<p>Update <code>cargo-tarpaulin@latest</code> to 0.35.4.</p>
</li>
<li>
<p>Update <code>cargo-sort@latest</code> to 2.1.4.</p>
</li>
</ul>
<h2>2.75.22</h2>
<ul>
<li>
<p>Update <code>tombi@latest</code> to 0.9.22.</p>
</li>
<li>
<p>Update <code>biome@latest</code> to 2.4.13.</p>
</li>
</ul>
<h2>2.75.21</h2>
<ul>
<li>
<p>Update <code>mise@latest</code> to 2026.4.19.</p>
</li>
<li>
<p>Update <code>tombi@latest</code> to 0.9.21.</p>
</li>
<li>
<p>Update <code>syft@latest</code> to 1.43.0.</p>
</li>
</ul>
<h2>2.75.20</h2>
<ul>
<li>
<p>Update <code>prek@latest</code> to 0.3.10.</p>
</li>
<li>
<p>Update <code>cargo-xwin@latest</code> to 0.22.0.</p>
</li>
</ul>
<h2>2.75.19</h2>
<ul>
<li>
<p>Update <code>wasmtime@latest</code> to 44.0.0.</p>
</li>
<li>
<p>Update <code>tombi@latest</code> to 0.9.20.</p>
</li>
<li>
<p>Update <code>martin@latest</code> to 1.6.0.</p>
</li>
<li>
<p>Update <code>just@latest</code> to 1.50.0.</p>
</li>
<li>
<p>Update <code>mise@latest</code> to 2026.4.18.</p>
</li>
<li>
<p>Update <code>rclone@latest</code> to 1.73.5.</p>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/taiki-e/install-action/blob/main/CHANGELOG.md">taiki-e/install-action's
changelog</a>.</em></p>
<blockquote>
<h1>Changelog</h1>
<p>All notable changes to this project will be documented in this
file.</p>
<p>This project adheres to <a href="https://semver.org">Semantic
Versioning</a>.</p>
<!-- raw HTML omitted -->
<h2>[Unreleased]</h2>
<ul>
<li>
<p>Update <code>prek@latest</code> to 0.3.11.</p>
</li>
<li>
<p>Update <code>mise@latest</code> to 2026.4.23.</p>
</li>
<li>
<p>Update <code>vacuum@latest</code> to 0.26.3.</p>
</li>
</ul>
<h2>[2.75.23] - 2026-04-27</h2>
<ul>
<li>
<p>Update <code>vacuum@latest</code> to 0.26.2.</p>
</li>
<li>
<p>Update <code>tombi@latest</code> to 0.9.24.</p>
</li>
<li>
<p>Update <code>mise@latest</code> to 2026.4.22.</p>
</li>
<li>
<p>Update <code>martin@latest</code> to 1.7.0.</p>
</li>
<li>
<p>Update <code>git-cliff@latest</code> to 2.13.1.</p>
</li>
<li>
<p>Update <code>cargo-tarpaulin@latest</code> to 0.35.4.</p>
</li>
<li>
<p>Update <code>cargo-sort@latest</code> to 2.1.4.</p>
</li>
</ul>
<h2>[2.75.22] - 2026-04-25</h2>
<ul>
<li>
<p>Update <code>tombi@latest</code> to 0.9.22.</p>
</li>
<li>
<p>Update <code>biome@latest</code> to 2.4.13.</p>
</li>
</ul>
<h2>[2.75.21] - 2026-04-24</h2>
<ul>
<li>
<p>Update <code>mise@latest</code> to 2026.4.19.</p>
</li>
<li>
<p>Update <code>tombi@latest</code> to 0.9.21.</p>
</li>
<li>
<p>Update <code>syft@latest</code> to 1.43.0.</p>
</li>
</ul>
<h2>[2.75.20] - 2026-04-23</h2>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/taiki-e/install-action/commit/481c34c1cf3a84c68b5e46f4eccfc82af798415a"><code>481c34c</code></a>
Release 2.75.23</li>
<li><a
href="https://github.com/taiki-e/install-action/commit/9109405c61e64f1e88e55c82819a46c97e835f4c"><code>9109405</code></a>
Update changelog</li>
<li><a
href="https://github.com/taiki-e/install-action/commit/cfd6b0ff1e194a5fc4fd4f692b7f06903f648672"><code>cfd6b0f</code></a>
Update <code>vacuum@latest</code> to 0.26.2</li>
<li><a
href="https://github.com/taiki-e/install-action/commit/b2ea47bca26c10b3e7128c6d89359c70fa463469"><code>b2ea47b</code></a>
Update uv manifest</li>
<li><a
href="https://github.com/taiki-e/install-action/commit/298078d06022bb442ff3a446a0387d1e51175e59"><code>298078d</code></a>
Update <code>tombi@latest</code> to 0.9.24</li>
<li><a
href="https://github.com/taiki-e/install-action/commit/4c81eb55c75148bda1b398a2c8fe099617e19bec"><code>4c81eb5</code></a>
Update prek manifest</li>
<li><a
href="https://github.com/taiki-e/install-action/commit/bb5f177651d41c019b21549ec8b706eb08ecf371"><code>bb5f177</code></a>
Update <code>mise@latest</code> to 2026.4.22</li>
<li><a
href="https://github.com/taiki-e/install-action/commit/c630b05e422c987c36e541b211d0330d66f20bcc"><code>c630b05</code></a>
Update <code>martin@latest</code> to 1.7.0</li>
<li><a
href="https://github.com/taiki-e/install-action/commit/b12e2ca23107113b92a8280c40d7ab6d82318049"><code>b12e2ca</code></a>
Update <code>git-cliff@latest</code> to 2.13.1</li>
<li><a
href="https://github.com/taiki-e/install-action/commit/3a693fd6338bba43206ee4454d311fc4160872f3"><code>3a693fd</code></a>
Update <code>cargo-tarpaulin@latest</code> to 0.35.4</li>
<li>Additional commits viewable in <a
href="https://github.com/taiki-e/install-action/compare/055f5df8c3f65ea01cd41e9dc855becd88953486...481c34c1cf3a84c68b5e46f4eccfc82af798415a">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=taiki-e/install-action&package-manager=github_actions&previous-version=2.75.18&new-version=2.75.23)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot
chore(deps): update pydata-sphinx-theme requirement from <1,>=0.17.0 …
288a5f1 
…to >=0.17.1,<1 in /docs (apache#21889)

Updates the requirements on
[pydata-sphinx-theme](https://github.com/pydata/pydata-sphinx-theme) to
permit the latest version.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/pydata/pydata-sphinx-theme/releases">pydata-sphinx-theme's
releases</a>.</em></p>
<blockquote>
<h2>v0.17.1</h2>
<h2>What's Changed</h2>
<h3>Fixes</h3>
<ul>
<li>Fix docs build with sphinx 'extensions' config key by <a
href="https://github.com/Yann-P"><code>@​Yann-P</code></a> in <a
href="https://redirect.github.com/pydata/pydata-sphinx-theme/pull/2352">pydata/pydata-sphinx-theme#2352</a></li>
<li>Fix default behavior when shorten_urls is unset by <a
href="https://github.com/hntk03"><code>@​hntk03</code></a> in <a
href="https://redirect.github.com/pydata/pydata-sphinx-theme/pull/2351">pydata/pydata-sphinx-theme#2351</a></li>
<li>Fix search input receiving keystrokes after escape dismiss in Safari
by <a href="https://github.com/tmchow"><code>@​tmchow</code></a> in <a
href="https://redirect.github.com/pydata/pydata-sphinx-theme/pull/2338">pydata/pydata-sphinx-theme#2338</a></li>
<li>Replace broken Unsplash image URLs with Picsum in light-dark guide
by <a href="https://github.com/tmchow"><code>@​tmchow</code></a> in <a
href="https://redirect.github.com/pydata/pydata-sphinx-theme/pull/2341">pydata/pydata-sphinx-theme#2341</a></li>
</ul>
<h3>Improvements</h3>
<ul>
<li>feat: add disable_search theme option by <a
href="https://github.com/PhilipSchmid"><code>@​PhilipSchmid</code></a>
in <a
href="https://redirect.github.com/pydata/pydata-sphinx-theme/pull/2315">pydata/pydata-sphinx-theme#2315</a></li>
</ul>
<h3>Dependencies</h3>
<ul>
<li>Bump pydata/pydata-sphinx-theme/.github/workflows/docs.yml from
43f9003549c495bcb43483fa21b6c1c396247f39 to
cd00a86a87aba3903543c00c5ce5f009c436e0b1 by <a
href="https://github.com/dependabot"><code>@​dependabot</code></a>[bot]
in <a
href="https://redirect.github.com/pydata/pydata-sphinx-theme/pull/2342">pydata/pydata-sphinx-theme#2342</a></li>
<li>Bump pydata/pydata-sphinx-theme/.github/workflows/CI.yml from
e8db643b990df73812cf9397bc0f8cfa1164e4d3 to
cd00a86a87aba3903543c00c5ce5f009c436e0b1 by <a
href="https://github.com/dependabot"><code>@​dependabot</code></a>[bot]
in <a
href="https://redirect.github.com/pydata/pydata-sphinx-theme/pull/2345">pydata/pydata-sphinx-theme#2345</a></li>
<li>Bump octokit/request-action from 2.4.0 to 3.0.0 by <a
href="https://github.com/dependabot"><code>@​dependabot</code></a>[bot]
in <a
href="https://redirect.github.com/pydata/pydata-sphinx-theme/pull/2346">pydata/pydata-sphinx-theme#2346</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a
href="https://github.com/PhilipSchmid"><code>@​PhilipSchmid</code></a>
made their first contribution in <a
href="https://redirect.github.com/pydata/pydata-sphinx-theme/pull/2315">pydata/pydata-sphinx-theme#2315</a></li>
<li><a href="https://github.com/tmchow"><code>@​tmchow</code></a> made
their first contribution in <a
href="https://redirect.github.com/pydata/pydata-sphinx-theme/pull/2338">pydata/pydata-sphinx-theme#2338</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/pydata/pydata-sphinx-theme/compare/v0.17.0...v0.17.1">https://github.com/pydata/pydata-sphinx-theme/compare/v0.17.0...v0.17.1</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/pydata/pydata-sphinx-theme/commit/a4986faa4b8241e041634d39ca6ed0e30c175240"><code>a4986fa</code></a>
bump: 0.17.0 → 0.17.1 (<a
href="https://redirect.github.com/pydata/pydata-sphinx-theme/issues/2355">#2355</a>)</li>
<li><a
href="https://github.com/pydata/pydata-sphinx-theme/commit/ff8be647971c91f759ef20222355ea57cc39362f"><code>ff8be64</code></a>
Fix default behavior when shorten_urls is unset (<a
href="https://redirect.github.com/pydata/pydata-sphinx-theme/issues/2351">#2351</a>)</li>
<li><a
href="https://github.com/pydata/pydata-sphinx-theme/commit/086775948a2e98ede34ccc0568508127eac5dc92"><code>0867759</code></a>
Replace broken Unsplash image URLs with Picsum in light-dark guide (<a
href="https://redirect.github.com/pydata/pydata-sphinx-theme/issues/2341">#2341</a>)</li>
<li><a
href="https://github.com/pydata/pydata-sphinx-theme/commit/56157e947983f5dee69650619e4c991ebe1016ba"><code>56157e9</code></a>
Bump octokit/request-action from 2.4.0 to 3.0.0 (<a
href="https://redirect.github.com/pydata/pydata-sphinx-theme/issues/2346">#2346</a>)</li>
<li><a
href="https://github.com/pydata/pydata-sphinx-theme/commit/8cce21de32c72ae5ab063cf14152ea17704b66c8"><code>8cce21d</code></a>
Bump pydata/pydata-sphinx-theme/.github/workflows/CI.yml from
e8db643b990df73...</li>
<li><a
href="https://github.com/pydata/pydata-sphinx-theme/commit/30b4830ab76f28b4e4e752ddd82e589cfef59a22"><code>30b4830</code></a>
Bump pydata/pydata-sphinx-theme/.github/workflows/docs.yml from
43f9003549c49...</li>
<li><a
href="https://github.com/pydata/pydata-sphinx-theme/commit/6673b56f499fa54c40c2c0c828ca21ba3f0d40c1"><code>6673b56</code></a>
Fix docs build with sphinx 'extensions' config key (<a
href="https://redirect.github.com/pydata/pydata-sphinx-theme/issues/2352">#2352</a>)</li>
<li><a
href="https://github.com/pydata/pydata-sphinx-theme/commit/ebc3f1723a51750e7e3bc3b0851d7fa88a467c83"><code>ebc3f17</code></a>
Fix search input receiving keystrokes after escape dismiss in Safari (<a
href="https://redirect.github.com/pydata/pydata-sphinx-theme/issues/2338">#2338</a>)</li>
<li><a
href="https://github.com/pydata/pydata-sphinx-theme/commit/8e676945c216502bc883baf1ccd4ef70244696f7"><code>8e67694</code></a>
feat: add disable_search theme option (<a
href="https://redirect.github.com/pydata/pydata-sphinx-theme/issues/2315">#2315</a>)</li>
<li>See full diff in <a
href="https://github.com/pydata/pydata-sphinx-theme/compare/v0.17.0...v0.17.1">compare
view</a></li>
</ul>
</details>
<br />

<details>
<summary>Most Recent Ignore Conditions Applied to This Pull
Request</summary>

| Dependency Name | Ignore Conditions |
| --- | --- |
| pydata-sphinx-theme | [>= 0.16.dev0, < 0.17] |
</details>


Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot
chore(deps): bump libloading from 0.8.9 to 0.9.0 (apache#21890)
53bd344 
Bumps [libloading](https://github.com/nagisa/rust_libloading) from 0.8.9
to 0.9.0.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/nagisa/rust_libloading/commit/dab97c569b33bd515e16637b8dedbdc696d9ec9c"><code>dab97c5</code></a>
0.9.0 release rites</li>
<li><a
href="https://github.com/nagisa/rust_libloading/commit/e89380446ed04ff14fe2f8adbd8cabc82ea54ad4"><code>e893804</code></a>
refactor: simplify null checks, remove some iffy implementations</li>
<li><a
href="https://github.com/nagisa/rust_libloading/commit/4b98285041defabf87df73f57051f5d55125758f"><code>4b98285</code></a>
add no-std support</li>
<li>See full diff in <a
href="https://github.com/nagisa/rust_libloading/compare/0.8.9...0.9.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=libloading&package-manager=cargo&previous-version=0.8.9&new-version=0.9.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@Jefffrey
refactor array_remove benchmarks & add nested benches (apache#21834)
4876cdc 
## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->

N/A

## Rationale for this change

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->

Looking to push a PR on optimizing `array_remove` later; discovered we
were lacking coverage for `array_remove_n` and `array_remove_all`
variants, as well as nested types. In the process of adding them,
decided to also do some cleanup of the benchmarks, specifically to
generate data in a more targeted fashion suitable to what we're
benchmarking in the functions (removing a specified needle from a
haystack).

## What changes are included in this PR?

<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

- Refactor functions used to create test data, unifying where possible
- Remove f64, decimal & binary benchmarks
- Add nested i64 (list<list<i64>>) benchmarks
- Add remove_n & remove_all benchmarks for i64 and nested i64

## Are these changes tested?

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

Benchmark only changes.

## Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

No.

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
@neilconway
perf: Use bulk-NULL builder in replace (apache#21849)
720aaff 
## Which issue does this PR close?

- Closes apache#21848.

## Rationale for this change

We can use the new bulk-NULL string builder APIs to reduce NULL handling
overhead in `replace`. There is a further opportunity for optimization
once arrow/arrow-rs#9692 lands.

Benchmarks:

- `replace size=1024/replace_string_ascii_single [str_len=32]`: 17.42 µs
→ 14.35 µs, **−17.71%**
- `replace size=1024/replace_string_view [str_len=32]`: 21.23 µs → 18.89
µs, **−11.05%**
- `replace size=1024/replace_string [str_len=32]`: 20.42 µs → 18.51 µs,
**−9.36%**
- `replace size=1024/replace_large_string [str_len=32]`: 20.83 µs →
18.42 µs, **−11.60%**
- `replace size=1024/replace_string_ascii_single [str_len=128]`: 10.61
µs → 10.37 µs, **−2.22%**
- `replace size=1024/replace_string_view [str_len=128]`: 28.37 µs →
28.03 µs, **−1.21%**
- `replace size=1024/replace_string [str_len=128]`: 27.96 µs → 28.03 µs,
**+0.25%** (noise)
- `replace size=1024/replace_large_string [str_len=128]`: 28.15 µs →
27.95 µs, **−0.70%** (noise)
- `replace size=4096/replace_string_ascii_single [str_len=32]`: 68.40 µs
→ 56.15 µs, **−17.94%**
- `replace size=4096/replace_string_view [str_len=32]`: 85.95 µs → 76.84
µs, **−10.61%**
- `replace size=4096/replace_string [str_len=32]`: 82.79 µs → 74.71 µs,
**−9.76%**
- `replace size=4096/replace_large_string [str_len=32]`: 85.32 µs →
75.14 µs, **−11.94%**
- `replace size=4096/replace_string_ascii_single [str_len=128]`: 38.35
µs → 39.10 µs, **+1.96%** (small regression)
- `replace size=4096/replace_string_view [str_len=128]`: 132.77 µs →
128.08 µs, **−3.53%**
- `replace size=4096/replace_string [str_len=128]`: 127.71 µs → 128.29
µs, **+0.46%** (noise)
- `replace size=4096/replace_large_string [str_len=128]`: 131.97 µs →
128.68 µs, **−2.49%**

## What changes are included in this PR?

* Use bulk-NULL string builders in `replace`

## Are these changes tested?

Yes, covered by existing tests.

## Are there any user-facing changes?

No.
@comphead @rluvaton
feat: minor lambda perf improvements (apache#21896)
8f033e4 
## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->

- Closes #.

Followup on
apache#21679 (comment)

## Rationale for this change

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->

## What changes are included in this PR?

<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

## Are these changes tested?

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

## Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->

---------

Co-authored-by: Raz Luvaton <16746759+rluvaton@users.noreply.github.com>
@alamb
Update astral-tokio-tar to appease cargo_audit (apache#21902)
b9cf885 
## Which issue does this PR close?


- Closes apache#21901

## Rationale for this change

Get a clean CI

## What changes are included in this PR?

Run 
```shell
cargo update -p astral-tokio-tar
```

And then check in the results

## Are these changes tested?

By CI

## Are there any user-facing changes?

No (and this library is used in testing, not in DataFusion)
@Jefffrey
feat: automatically cast ListView to List for UDFs (apache#21855)
e8a93bb 
## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->

- Part of apache#21777

## Rationale for this change

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->

First step in supporting `ListView` types in UDFs. Previously the UDFs
would error if trying to pass a `ListView` type to nested UDFs. Now we
can pass in `ListView` inputs and they are cast to `List`/`LargeList`
(similar to `FixedSizeList` if that coercion mode is enabled).

## What changes are included in this PR?

<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

In type coercion, for array signatures automatically coerce any
`ListView`/`LargeListView` types to `List`/`LargeList` respectively.
Also add `ListView` support to various array util functions.

## Are these changes tested?

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

Added SLTs.

## Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

No.

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
@gabotechs
Remove unnecessary Mutex in SharedMemoryReservation (apache#21899)
61fe692 
## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->

- Closes #.

## Rationale for this change

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->

After apache#19759, all the methods
of a `MemoryReservation` take &self instead of &mut self, so no need to
hide a `SharedMemoryReservation` behind a Mutex.

## What changes are included in this PR?

<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

Remove an unnecessary Mutex

## Are these changes tested?

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

By existing unit tests

## Are there any user-facing changes?

No
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
@rluvaton @claude @mbutrovich
ci: add breaking change detector (apache#21499)
73ca6a5 
## Which issue does this PR close?

Partially closes:
- apache#21406

## Rationale for this change

detect breaking changes

## What changes are included in this PR?

add new github workflow 

## Are these changes tested?

Looks like it is working

## Are there any user-facing changes?
no

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Matt Butrovich <mbutrovich@users.noreply.github.com>
@fred1268
fix: fix elapsed_compute metric in ParquetSink to report encoding tim…
3aefba7 
…e only (apache#21825)

## Which issue does this PR close?

- Closes apache#21797.

## Rationale for this change

`ParquetSink` registered an `elapsed_compute` metric using a single
wall-clock timer that spanned the entire write operation — upstream
batch wait, CPU Arrow→Parquet encoding, and object-store I/O all rolled
into one number. This
made the metric misleading: it inflated `elapsed_compute` with I/O
latency, which is inconsistent with how every other operator in
DataFusion reports this metric (CPU time only).

## What changes are included in this PR?

Two write paths are fixed independently:

**Sequential path** (`allow_single_file_parallelism = false` or CDC
enabled):
- A new `TimingWriter<W>` wrapper implements `AsyncFileWriter` and
records wall-clock time spent in I/O calls (`write` / `complete`).
- The total time inside `writer.write()` and `writer.close()` is
accumulated in `total_write_time`. After all tasks join,
`elapsed_compute` is set to `total_write_time − io_time`, isolating pure
Arrow→Parquet encoding time.

**Parallel path** (`allow_single_file_parallelism = true`, default):
- `encoding_time: Time` (a clone of the registered `elapsed_compute`
metric) is threaded through the five-function call chain down to the two
leaf sites: `writer.write()` in `column_serializer_task` and
`writer.close()` in `spawn_rg_join_and_finalize_task`. Since `Time` is
`Arc<AtomicUsize>`, all concurrent column tasks accumulate directly into
the registered metric.
- Note: on the parallel path, `append_to_row_group()` in
`concatenate_parallel_row_groups` is interleaved with I/O and cannot be
cleanly isolated. It is excluded from `elapsed_compute`. This is
acceptable since it operates on already-encoded data and represents a
small fraction of total encoding CPU time.

## Are these changes tested?

Yes. Two tests are added/extended in
`datafusion/core/src/dataframe/parquet.rs`:
- `test_parquet_sink_metrics_sequential` (new): verifies
`elapsed_compute > 0` with `allow_single_file_parallelism = false`.
- `test_parquet_sink_metrics_parallel`: extended with an
`elapsed_compute > 0` assertion (was previously missing for the parallel
path).

The existing `test_parquet_sink_metrics` test (parallel path, default
config) already asserted `elapsed_compute > 0` and continues to pass.

## Are there any user-facing changes?

No API changes. The `elapsed_compute` metric was already surfaced — this
PR makes its value accurate rather than introducing a new metric.
@Jefffrey
Fix GH action permissions in rust.yml and docs.yaml workflows (ap…
66980e2 
…ache#21884)

## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->

Followup to apache#21838 for the rest
of the workflows

## Rationale for this change

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->

## What changes are included in this PR?

<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

## Are these changes tested?

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

## Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
geoffreyclaude and others added 15 commits June 24, 2026 13:14
@geoffreyclaude
Merge pull request #149 from DataDog/codex/branch-54-proto-downcast-d…
c884d40 
…elegate

fix(proto): honor ExecutionPlan downcast_delegate during serialization
@fred1268
Merge pull request #138 from DataDog/cherry-pick-434957ec8
48ad4c2 
Cherry pick 434957e - turn off submodule updating
@fred1268
Merge pull request #139 from DataDog/cherry-pick-a0763db98
3df09c5 
Cherry pick a0763db - Fix ArrayCompact incompatibility
@fred1268
Merge pull request #140 from DataDog/cherry-pick-6692f6f6d
f353729 
Cherry pick 6692f6f - fix(substrait): dedupe names
@fred1268
Merge pull request #141 from DataDog/cherry-pick-12d6c8125
46dcff3 
Cherry pick 12d6c81 - Add lambda substrait support (apache#21193) (#134)
@fred1268
Merge pull request #142 from DataDog/cherry-pick-b5022000c
094ad69 
Cherry pick b502200 - fix: make PushDownLeafProjections work with unnest
@fred1268
Merge pull request #146 from DataDog/cherry-pick-93ba65a
04d591e 
Cherry pick 93ba65a - add downcast_delegate
@mkleen @fred1268
fix: Avoid panicing when stats are not available for a file group spl…
ff017d0 
…it (apache#23277)

## Which issue does this PR close?

- Closes apache#23219.

## Rationale for this change

The query from the issue: 

```sql
SELECT   (((Cast(id AS BIGINT) % 1024) + 1024) % 1024) AS computed_bucket
FROM     profile
ORDER BY computed_bucket,
         Cast(id AS BIGINT) limit 10;
```

panics:

```
thread 'main' panicked at .../datafusion-datasource-54.0.0/src/statistics.rs:100:48:
index out of bounds: the len is 0 but the index is 0
```

The underlying issue is that the current code panics when files are
split by statistics and there are no statistics available for the column
where the sort order is defined in this case `computed_bucket`.

## What changes are included in this PR?

- Fix in `MinMaxStatistics` to check if there are stats available for a
given column
- Test

## Are these changes tested?

Yes

## Are there any user-facing changes?

No
@fred1268
Merge pull request #153 from DataDog/cherry-pick-7b1bffa
b523ae4 
Cherry pick 7b1bffa - fix: Avoid panicing when stats are not available for a file group spl…
@Adam-Alani @alamb
Cherry-pick apache#22744 and apache#23312 (#157)
4a6f1ba 
* chore: Update to arrow/parquet 59.1.0 (apache#23312)

(DRAFT until arrow is updated, I am using this PR to pre-test the
release)

- related to apache/arrow-rs#9878

Update to latest arrow

Yes by CI

No API change (this is a minor update of Arrow)

(cherry picked from commit f34a676)

* chore: Update to arrow/parquet 59.0.0 (apache#22744)

- related to apache/arrow-rs#9110

Update to latest version of arrow/parquet

1. Update to arrow/parquet 59.0.0
2. Adjust code for API differences

By CI

New dependency

(cherry picked from commit dae03ee)

---------

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
@gstvg @gabotechs
Add protobuf support for lambdas (apache#22362)
9d2114f 
## Which issue does this PR close?

Part of apache#21172

## Rationale for this change

Protobuf support wasn't implemented in main lambda PR to not make it
even bigger

## What changes are included in this PR?

Protobuf encoding and decoding (~1000 LOC in generated files, ~210 impl,
~400 tests)

## Are these changes tested?

Unit tests, similar to the existing ones for scalar functions

## Are there any user-facing changes?

Proto `ExprType` has new variants
@Tristan1900 @gabotechs
[branch-54] Adapt lambda protobuf backport
b22414e 
Update the pre-refactor upstream implementation for the concrete HigherOrderUDF wrapper already present on branch-54. Preserve the DynamicFilterPhysicalExpr import required by existing branch-54 tests and regenerate protobuf JSON bindings with the branch-54 generator.
@fred1268 @alamb
perf: array_agg() performance improvements (apache#23716)
0fccf15 
- Closes apache#23715.

Queries using `array_agg(DISTINCT col)` were significantly slower than
expected.

Profiling revealed that `DistinctArrayAggAccumulator::update_batch` was
allocating a
heap-owned `String` on every single input row — even for rows whose
value was already
present in the accumulator. For a typical low-cardinality workload (e.g.
a column of ~25
database names across thousands of rows), this meant paying the full
allocation cost for
every duplicate, which dominated the runtime.

This PR applies the same deduplication strategy already used by
`AggregateExec` for
`GROUP BY`: duplicate rows now cost only a hash probe with no heap
allocation, and new
distinct values are appended to a single shared buffer rather than
allocated individually.
The fix applies to all column types, not just strings, and
`retract_batch` support
(required for sliding window frames such as `ROWS BETWEEN N PRECEDING
AND CURRENT ROW`)
is fully preserved.

Four unit tests were added to `DistinctArrayAggAccumulator` — one each
for `Utf8`,
`Int64`, `Float64`, and `Date32` — to pin the deduplication contract
across the most
common column types and serve as a regression guard for future changes.
The existing
sliding window sqllogictest suite (`array_agg_sliding_window.slt`)
covers `retract_batch`
correctness end-to-end and passes unchanged.

Two `update_batch` micro-benchmarks were added to measure the
before/after on realistic
data: one with low cardinality (~25 distinct database names in 8 192
rows, modelling the
common production case) and one with high cardinality (~7 800 distinct
values, modelling
the worst case where almost every row is new). Results on an 8 192-row
batch:

| Benchmark | Before | After | Speedup |
|---|---|---|---|
| Low cardinality (~25 distinct DB names) | 648.6 µs | 189.4 µs |
**3.42×** |
| High cardinality (~7 800 distinct values) | 1078.3 µs | 269.4 µs |
**4.00×** |

No user-facing changes

No breaking changes to public APIs

---------

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
@fred1268
cargo fmt
dbbd836
@gh-worker-dd-mergequeue-cf854d @fred1268
Merge pull request #161 from DataDog/cherry-pick-cd28203e3
b632fb7 
Cherry-pick-cd28203e3 - perf: `array_agg()` performance improvements (apache#23716)

Co-authored-by: fred1268 <frederic.thomas@datadoghq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

catalog common core development-process documentation execution functions logical-expr optimizer physical-expr proto sql sqllogictest substrait

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.