Repository navigation
Conversation
rich7420
force-pushed
the
feat/2837-native-map-explode
branch
from
September 28, 2026 14:50
76768f2 to
9601e81
Compare
Cover JVM-fed maps, nullable Boolean batch boundaries, all-NULL/empty outer batches, native map producers, and stored floating-point keys. Require native generators and matching Spark results and schemas. Benchmark map variants and nested map/array inputs with sinks that consume keys and positions. Remove the unrelated Parquet field-ID assertion change so it can be reviewed independently.
Keep input arrays as temporary placeholders until generated columns replace them. Build take indices only when a carried column consumes them. Cover recursive unnest with and without carried columns and both drop and outer null handling. Preserve the existing bounded map and array execution path.
rich7420
marked this pull request as ready for review
October 3, 2026 16:36
Port map planning to the extracted ExplodeBuilder while preserving once-per-batch evaluation and returning an error for malformed map entry types.
Split individual List rows before materialization and cap repetition so carried buffers and child offsets stay within Arrow's offset limits. Preserve shared entry buffers, positions and outer semantics.
Fill non-null primitive runs directly for expansions of at least five and build short-string offsets with one Vec extension per run. Retain Arrow take for smaller expansions and nullable primitives.
Native generators materialize carried columns between expansions and were slower than Spark's pipeline in the nested benchmark. Keep maps followed by another generator on Spark across projections and filters, while retaining native single maps and chains separated by an exchange.
Keep normal batches intact and skip offset checks when repetition is already safe. Remove the custom carried-value kernels, retain Arrow take, and avoid unused allocations when no columns are carried. Restore array and map benchmark coverage and consolidate map SQL cases.
Sliced list and map arrays retain their full child arrays, which inflates Arrow take's allocation estimate. Trim the referenced child span before taking carried columns, preserving nested metadata and validity.
Remove the map-only planner fallback so maps and arrays share the bounded native explode path. Cover both chain directions across all generator variants and AQE, add the reverse-chain benchmark, and update the SQL expectation and support notes.
Replace the 2 GiB carried-string stream test with real-boundary helper checks and small-limit stream cases for Utf8, Binary and Struct values. Reuse private stream construction and pass its offset limit through the existing chunk guards. Check batching, output values and retained buffers, including skewed rows that should remain in one batch.
…get (apache#7482) GitHub blocks pull_request_target on public repositories from December 2, 2026, and ASF is disabling it. A no-op pull_request workflow now starts the labeler, which runs on workflow_run from the default branch, finds the pull request from the run's head repository, branch and commit, and applies the type and path labels through the API.
* chore: remove dead code from the release build Small cleanup to remove dead code from the release build script and the files it drives. * chore: address review feedback on the release build cleanup Remove the core-amd64, core-arm64 and release-linux Makefile targets, which nothing calls (the first two also end with a script that does not exist), the darwin resource entries in spark/pom.xml that only the removed cross-build produced, and the now-unneeded Dockerfile stage name.
* fix: package native dependency license materials Include reviewed native dependency license texts and notices in shaded artifacts and validate the generated bundle and packaged legal entries. Co-authored-by: Codex <codex@openai.com> * fix: scope native attribution to packaged platforms Select shared and platform-specific legal materials from actual native payloads, review code-generation output separately from host tooling, consolidate Apache terms with pinned exception review, and curate required notices. Co-authored-by: Codex <codex@openai.com> * fix: simplify native license packaging Replace the broad dependency inventory and generation tooling with a maintained binary LICENSE for the confirmed lz4_flex and Zstandard omissions. Check the shaded main and sources JARs using the existing Maven verification path. Co-authored-by: Codex <codex@openai.com> * docs: remove binary license section from release guide Remove the Binary license materials section while preserving the binary LICENSE, Maven packaging and artifact checks. Co-authored-by: Codex <codex@openai.com> * fix: validate native license versions and source jar options Check the two bundled compression license versions against Cargo.lock. Make the sources-JAR check configurable, report a missing expected source artifact explicitly, and honor maven.source.skip without skipping main-JAR checks. Co-authored-by: Codex <codex@openai.com> * fix: scope source-only JAR exceptions to sources artifacts Keep the binary allowlist strict while checking the explicit sources JAR with its three additional source paths. Track artifact roles by position so supplying the same file in both roles does not relax the binary check. Add paired artifact regression tests to the existing CI preflight. Co-authored-by: Codex <codex@openai.com> * fix: list each JAR once and test every license check The checker listed each artifact three times, and the two listings added for the native LICENSE check sat inside `if` and `$(...)`, where a jar tf failure never reached the SIGUSR1 trap. Read the listing once into a variable, fail explicitly when jar tf fails, and run all three checks on it. The native-library probe uses a here-string because grep -q on a pipe exits early and, under pipefail, can hide the match. The tests covered only the source-only allowlist. Add tests for missing, duplicate, truncated and Apache-only LICENSE entries, removed copyright lines, non-native JARs, lz4_flex and Zstandard drift from Cargo.lock, unreadable lock versions, a missing or skipped sources JAR, an unlistable JAR and a listing larger than a pipe buffer. Each test runs a copy of the checker against a fixture Cargo.lock, so a dependency bump no longer breaks unrelated tests. One test still runs the checker against the repository's lock file, so a bump of lz4_flex, or of zstd-sys to a new Zstandard, fails Preflight on purpose until LICENSE-binary.txt is updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Codex <codex@openai.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
CI mirror of upstream apache/datafusion-comet#6107, addressing apache#2837. This fork PR does not close or merge the upstream issue.
Rationale for this change
Map inputs to
explodeandposexplodecurrently fall back to Spark.What changes are included in this PR?
Support maps in all four explode variants by sharing their entry buffers with the existing unnest path. Preserve field metadata, entry order and outer NULL rows. Add regression tests and update support documentation.
How are these changes tested?
Local verification passed 18 Rust tests and 52 targeted tests on each of Spark 3.5.9 and 4.1.3, covering native execution, NULL/empty maps, nested inputs, sliced arrays and batch boundaries.
Upstream CI passed at
9601e81ba, including Required Checks. The corresponding fork run is not fully green: Spark 3.4 scans faileda file without ids next to a file with ids is checked on its ownin ParquetReadSuite (544 passed, one failed in that job). Its missing-field-ID error assertion failed. The relationship to this map-explode change has not been established; do not treat the run as a complete cross-profile pass until the failure is classified and resolved.