Repository navigation
Conversation
Decline FLOAT and DOUBLE range comparisons and zero or NaN literals. Spark treats -0.0 as equal to 0.0 and sorts every NaN above all numbers, while iceberg-rust compares by IEEE 754 total order, and its row-group pruning trusts statistics that can omit NaN (apache/iceberg-rust#3494). Decline equality under NOT on nullable columns. Iceberg passes `<=>` as `=`, so the native reader dropped NULL rows that Spark keeps for `NOT (id <=> 1)`. On columns whose type changed in the schema history, push only equality outside NOT and null checks, since iceberg-rust compares older, narrower files incorrectly (apache/iceberg-rust#3122). An incomplete history treats every BIGINT, DOUBLE and DECIMAL column as promoted. Closes apache#7136. Closes apache#7137. Part of apache#7138.
rich7420
force-pushed
the
feat/5883-iceberg-partial-pushdown
branch
from
October 22, 2026 05:52
bf663cf to
622d7e9
Compare
Spark rewrites `b = false` on a nullable boolean as `b IS NOT NULL AND NOT b`, which Iceberg passes as `not_null(b) AND not(b = true)`. Declining equality under NOT on a nullable column left that residual unpushed, so CometFuzzIcebergSuite found no residual for its boolean column. A NULL value fails a NOT_NULL conjunct, and with it the whole AND, in Spark and in the native reader alike. Treat a column checked that way as non-nullable within the AND, so the null-safe concern no longer applies.
…cans Extract referencedColumn so the residual converter and notNullColumns share the bare-reference check. Share the residual predicate loop of the new native tests, and assert that `id != 1` and `b = false` are still pushed while `NOT (id <=> 1)` and `b IS NOT TRUE` are not. CometFuzzIcebergSuite samples its equality literal from the data. Skip zero as well as NaN, since floating-point comparisons with zero are no longer pushed. Note in the Iceberg guide that `!=` is not affected by the negated-equality rule.
Retain convertible positive AND conjuncts when the rest of a residual is unsupported. OR still needs both branches, and NOT needs a complete child. Keep the full filter above the scan. For partial conversion, treat BIGINT and DOUBLE columns as potentially promoted and retain only null checks and equality outside NOT. Tables with a name mapping continue to require complete conversion. Add native pruning regressions and partial-filter benchmarks. Closes apache#5883.
Add a supported scan setting that disables partial residual conversion while preserving complete predicate pushdown. Verify both modes through the native scan and serialized residuals. Compare identical queries with partial pushdown enabled and disabled in the read benchmark, including a scattered 99% case and reversed ordering. Clarify the remaining whole-residual limitation for imported files.
rich7420
force-pushed
the
feat/5883-iceberg-partial-pushdown
branch
from
October 26, 2026 15:44
c742b46 to
d2bec64
Compare
…get (apache#7482) GitHub blocks pull_request_target on public repositories from December 2, 2026, and ASF is disabling it. A no-op pull_request workflow now starts the labeler, which runs on workflow_run from the default branch, finds the pull request from the run's head repository, branch and commit, and applies the type and path labels through the API.
* chore: remove dead code from the release build Small cleanup to remove dead code from the release build script and the files it drives. * chore: address review feedback on the release build cleanup Remove the core-amd64, core-arm64 and release-linux Makefile targets, which nothing calls (the first two also end with a script that does not exist), the darwin resource entries in spark/pom.xml that only the removed cross-build produced, and the now-unneeded Dockerfile stage name.
* fix: package native dependency license materials Include reviewed native dependency license texts and notices in shaded artifacts and validate the generated bundle and packaged legal entries. Co-authored-by: Codex <codex@openai.com> * fix: scope native attribution to packaged platforms Select shared and platform-specific legal materials from actual native payloads, review code-generation output separately from host tooling, consolidate Apache terms with pinned exception review, and curate required notices. Co-authored-by: Codex <codex@openai.com> * fix: simplify native license packaging Replace the broad dependency inventory and generation tooling with a maintained binary LICENSE for the confirmed lz4_flex and Zstandard omissions. Check the shaded main and sources JARs using the existing Maven verification path. Co-authored-by: Codex <codex@openai.com> * docs: remove binary license section from release guide Remove the Binary license materials section while preserving the binary LICENSE, Maven packaging and artifact checks. Co-authored-by: Codex <codex@openai.com> * fix: validate native license versions and source jar options Check the two bundled compression license versions against Cargo.lock. Make the sources-JAR check configurable, report a missing expected source artifact explicitly, and honor maven.source.skip without skipping main-JAR checks. Co-authored-by: Codex <codex@openai.com> * fix: scope source-only JAR exceptions to sources artifacts Keep the binary allowlist strict while checking the explicit sources JAR with its three additional source paths. Track artifact roles by position so supplying the same file in both roles does not relax the binary check. Add paired artifact regression tests to the existing CI preflight. Co-authored-by: Codex <codex@openai.com> * fix: list each JAR once and test every license check The checker listed each artifact three times, and the two listings added for the native LICENSE check sat inside `if` and `$(...)`, where a jar tf failure never reached the SIGUSR1 trap. Read the listing once into a variable, fail explicitly when jar tf fails, and run all three checks on it. The native-library probe uses a here-string because grep -q on a pipe exits early and, under pipefail, can hide the match. The tests covered only the source-only allowlist. Add tests for missing, duplicate, truncated and Apache-only LICENSE entries, removed copyright lines, non-native JARs, lz4_flex and Zstandard drift from Cargo.lock, unreadable lock versions, a missing or skipped sources JAR, an unlistable JAR and a listing larger than a pipe buffer. Each test runs a copy of the checker against a fixture Cargo.lock, so a dependency bump no longer breaks unrelated tests. One test still runs the checker against the repository's lock file, so a bump of lz4_flex, or of zstd-sys to a new Zstandard, fails Preflight on purpose until LICENSE-binary.txt is updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Codex <codex@openai.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
CI mirror for apache#6847.
Rationale for this change
Run the supplemental suite unavailable to the author through upstream labels.
What changes are included in this PR?
Uses the existing upstream PR branch at
aaf808d295992af3badee81691cf0d0e1ce6ec4dwithout additional code changes.How are these changes tested?
Upstream PR-tier CI has passed. Apply
run-iceberg-testshere for the supplemental suite; no run started. GitHub rejects manual dispatch with "Actions has been disabled for this repository" even though the settings API reports enabled. The upstream PR requests a maintainer to apply the matching label. This fork run tests the PR merged with fork main and does not replace upstream merge-queue checks.