Skip to content

perf: preserve supported Iceberg residual conjuncts - #48

Draft
rich7420 wants to merge 12 commits into
mainfrom
feat/5883-iceberg-partial-pushdown
Draft

rich7420 wants to merge 12 commits into
mainfrom
feat/5883-iceberg-partial-pushdown

Conversation

@rich7420

@rich7420 rich7420 commented Oct 17, 2026 •

Copy link
Copy Markdown
Owner

Which issue does this PR close?

CI mirror for apache#6847.

Rationale for this change

Run the supplemental suite unavailable to the author through upstream labels.

What changes are included in this PR?

Uses the existing upstream PR branch at aaf808d295992af3badee81691cf0d0e1ce6ec4d without additional code changes.

How are these changes tested?

Upstream PR-tier CI has passed. Apply run-iceberg-tests here for the supplemental suite; no run started. GitHub rejects manual dispatch with "Actions has been disabled for this repository" even though the settings API reports enabled. The upstream PR requests a maintainer to apply the matching label. This fork run tests the PR merged with fork main and does not replace upstream merge-queue checks.

Decline FLOAT and DOUBLE range comparisons and zero or NaN literals.
Spark treats -0.0 as equal to 0.0 and sorts every NaN above all
numbers, while iceberg-rust compares by IEEE 754 total order, and its
row-group pruning trusts statistics that can omit NaN
(apache/iceberg-rust#3494).

Decline equality under NOT on nullable columns. Iceberg passes `<=>`
as `=`, so the native reader dropped NULL rows that Spark keeps for
`NOT (id <=> 1)`.

On columns whose type changed in the schema history, push only
equality outside NOT and null checks, since iceberg-rust compares
older, narrower files incorrectly (apache/iceberg-rust#3122). An
incomplete history treats every BIGINT, DOUBLE and DECIMAL column as
promoted.

Closes apache#7136. Closes apache#7137. Part of apache#7138.
@rich7420
rich7420 force-pushed the feat/5883-iceberg-partial-pushdown branch from bf663cf to 622d7e9 Compare October 22, 2026 05:52
rich7420 and others added 6 commits October 22, 2026 22:43
Spark rewrites `b = false` on a nullable boolean as `b IS NOT NULL AND
NOT b`, which Iceberg passes as `not_null(b) AND not(b = true)`. Declining
equality under NOT on a nullable column left that residual unpushed, so
CometFuzzIcebergSuite found no residual for its boolean column.

A NULL value fails a NOT_NULL conjunct, and with it the whole AND, in
Spark and in the native reader alike. Treat a column checked that way as
non-nullable within the AND, so the null-safe concern no longer applies.
…cans

Extract referencedColumn so the residual converter and notNullColumns share
the bare-reference check. Share the residual predicate loop of the new native
tests, and assert that `id != 1` and `b = false` are still pushed while
`NOT (id <=> 1)` and `b IS NOT TRUE` are not.

CometFuzzIcebergSuite samples its equality literal from the data. Skip zero
as well as NaN, since floating-point comparisons with zero are no longer
pushed.

Note in the Iceberg guide that `!=` is not affected by the negated-equality
rule.
Retain convertible positive AND conjuncts when the rest of a residual is
unsupported. OR still needs both branches, and NOT needs a complete child.
Keep the full filter above the scan.

For partial conversion, treat BIGINT and DOUBLE columns as potentially
promoted and retain only null checks and equality outside NOT. Tables with
a name mapping continue to require complete conversion.

Add native pruning regressions and partial-filter benchmarks.

Closes apache#5883.
Add a supported scan setting that disables partial residual conversion
while preserving complete predicate pushdown. Verify both modes through
the native scan and serialized residuals.

Compare identical queries with partial pushdown enabled and disabled in
the read benchmark, including a scattered 99% case and reversed ordering.
Clarify the remaining whole-residual limitation for imported files.
@rich7420
rich7420 force-pushed the feat/5883-iceberg-partial-pushdown branch from c742b46 to d2bec64 Compare October 26, 2026 15:44
andygrove and others added 5 commits November 7, 2026 13:47
…get (apache#7482)

GitHub blocks pull_request_target on public repositories from December 2,
2026, and ASF is disabling it. A no-op pull_request workflow now starts the
labeler, which runs on workflow_run from the default branch, finds the pull
request from the run's head repository, branch and commit, and applies the
type and path labels through the API.
* chore: remove dead code from the release build

Small cleanup to remove dead code from the release build script and the
files it drives.

* chore: address review feedback on the release build cleanup

Remove the core-amd64, core-arm64 and release-linux Makefile targets, which
nothing calls (the first two also end with a script that does not exist),
the darwin resource entries in spark/pom.xml that only the removed
cross-build produced, and the now-unneeded Dockerfile stage name.
* fix: package native dependency license materials

Include reviewed native dependency license texts and notices in shaded artifacts and validate the generated bundle and packaged legal entries.

Co-authored-by: Codex <codex@openai.com>

* fix: scope native attribution to packaged platforms

Select shared and platform-specific legal materials from actual native payloads, review code-generation output separately from host tooling, consolidate Apache terms with pinned exception review, and curate required notices.

Co-authored-by: Codex <codex@openai.com>

* fix: simplify native license packaging

Replace the broad dependency inventory and generation tooling with a maintained binary LICENSE for the confirmed lz4_flex and Zstandard omissions. Check the shaded main and sources JARs using the existing Maven verification path.

Co-authored-by: Codex <codex@openai.com>

* docs: remove binary license section from release guide

Remove the Binary license materials section while preserving the binary LICENSE, Maven packaging and artifact checks.

Co-authored-by: Codex <codex@openai.com>

* fix: validate native license versions and source jar options

Check the two bundled compression license versions against Cargo.lock. Make the sources-JAR check configurable, report a missing expected source artifact explicitly, and honor maven.source.skip without skipping main-JAR checks.

Co-authored-by: Codex <codex@openai.com>

* fix: scope source-only JAR exceptions to sources artifacts

Keep the binary allowlist strict while checking the explicit sources JAR with
its three additional source paths. Track artifact roles by position so supplying
the same file in both roles does not relax the binary check.

Add paired artifact regression tests to the existing CI preflight.

Co-authored-by: Codex <codex@openai.com>

* fix: list each JAR once and test every license check

The checker listed each artifact three times, and the two listings added
for the native LICENSE check sat inside `if` and `$(...)`, where a jar
tf failure never reached the SIGUSR1 trap. Read the listing once into a
variable, fail explicitly when jar tf fails, and run all three checks on
it. The native-library probe uses a here-string because grep -q on a
pipe exits early and, under pipefail, can hide the match.

The tests covered only the source-only allowlist. Add tests for missing,
duplicate, truncated and Apache-only LICENSE entries, removed copyright
lines, non-native JARs, lz4_flex and Zstandard drift from Cargo.lock,
unreadable lock versions, a missing or skipped sources JAR, an
unlistable JAR and a listing larger than a pipe buffer.

Each test runs a copy of the checker against a fixture Cargo.lock, so a
dependency bump no longer breaks unrelated tests. One test still runs
the checker against the repository's lock file, so a bump of lz4_flex,
or of zstd-sys to a new Zstandard, fails Preflight on purpose until
LICENSE-binary.txt is updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants