Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
2dadfe5
feat: support bloom filters in native Iceberg writes
NikitaMatskevich Sep 5, 2026
cf12c5a
feat: support Iceberg bloom filter write settings
NikitaMatskevich Sep 6, 2026
657affc
fix: fall back for sanitized Iceberg bloom paths
NikitaMatskevich Sep 8, 2026
7309f2d
test: gate Iceberg bloom shape properties independently
NikitaMatskevich Sep 8, 2026
39c18e4
fix: match Iceberg-version bloom property support
NikitaMatskevich Sep 8, 2026
f07bd9a
fix: use Parquet 59 bloom filter APIs
NikitaMatskevich Sep 8, 2026
4c72413
fix: satisfy formatting checks
NikitaMatskevich Sep 8, 2026
927aaf9
docs: describe Parquet 59 bloom folding
NikitaMatskevich Sep 12, 2026
1058cdf
fix: reject zero bloom filter FPP
NikitaMatskevich Sep 12, 2026
505efce
fix: ignore unused bloom max byte caps
NikitaMatskevich Sep 12, 2026
534eabe
test: document bloom folding after writes
NikitaMatskevich Sep 12, 2026
98c4273
test: lock bloom sizing parity across runtimes
NikitaMatskevich Sep 12, 2026
9698a8d
refactor: group Iceberg bloom column settings
NikitaMatskevich Sep 13, 2026
866601d
fix: match Iceberg bloom path resolution by version
NikitaMatskevich Sep 13, 2026
ed9ba90
feat: make native Iceberg bloom folding opt-in
NikitaMatskevich Sep 14, 2026
8150a05
fix: integrate Bloom write opt-in with current Iceberg writer
NikitaMatskevich Oct 24, 2026
cb65a71
docs: clarify native Iceberg Bloom filter tradeoffs
NikitaMatskevich Oct 24, 2026
bf5f06a
docs: simplify Bloom filter compatibility introduction
NikitaMatskevich Oct 24, 2026
96edb6c
docs: list exceptional Bloom filter fallback cases
NikitaMatskevich Oct 24, 2026
a28b550
docs: remove upstream folding work from user guide
NikitaMatskevich Oct 24, 2026
e1f46ab
docs: group Bloom write prerequisites under accepted divergences
NikitaMatskevich Oct 24, 2026
b124dfb
fix: fall back for floating-point Iceberg Bloom writes
NikitaMatskevich Oct 25, 2026
c41a053
fix: ignore unconfigured columns in Iceberg Bloom path checks
NikitaMatskevich Oct 25, 2026
b8dfbe2
fix: fall back for unidentified Iceberg Bloom naming conventions
NikitaMatskevich Oct 25, 2026
14e23f4
docs: clarify Iceberg Bloom opt-in and unused caps
NikitaMatskevich Oct 25, 2026
2439939
test: cover Iceberg Bloom parity across physical types
NikitaMatskevich Oct 25, 2026
6915bc4
fix: bound Iceberg Bloom synthetic NDV search
NikitaMatskevich Oct 25, 2026
86c8a47
bench: compare native Iceberg Bloom writes
NikitaMatskevich Oct 25, 2026
264b6c1
bench: remove one-off Bloom measurement runner
NikitaMatskevich Oct 25, 2026
0e819d9
fix: limit native Bloom writes to modern Iceberg runtimes
NikitaMatskevich Oct 25, 2026
f98171a
fix: remove redundant interpolation in Bloom write test
NikitaMatskevich Oct 26, 2026
829d0e9
fix: remove unused Bloom test callback parameters
NikitaMatskevich Oct 27, 2026
26ed2e6
docs: align Bloom eligibility with supported Spark profiles
NikitaMatskevich Nov 10, 2026
89d8079
test: use physical Bloom names on Iceberg 1.5
NikitaMatskevich Oct 25, 2026
8fdeb4e
fix: normalize legacy Iceberg Bloom paths
NikitaMatskevich Oct 25, 2026
51d745f
fix: fall back for colliding legacy Bloom aliases
NikitaMatskevich Oct 25, 2026
dee1707
fix: preserve legacy Bloom lookup validation
NikitaMatskevich Oct 25, 2026
98c7cd3
fix: retain independent legacy Bloom FPP validation
NikitaMatskevich Oct 25, 2026
847d666
feat: enable safeguarded native Bloom writes on legacy Iceberg
NikitaMatskevich Oct 25, 2026
72837ef
test: cover legacy Bloom caps without NDV support
NikitaMatskevich Oct 25, 2026
9e14d3a
docs: describe native Bloom support on legacy Iceberg
NikitaMatskevich Oct 25, 2026
eb59283
docs: clarify Iceberg 1.6 and 1.7 Bloom properties
NikitaMatskevich Nov 9, 2026
b6d2f86
refactor: share configured Bloom column discovery
NikitaMatskevich Nov 9, 2026
65ecc43
refactor: report Bloom path fallback reasons directly
NikitaMatskevich Nov 9, 2026
220a532
docs: document legacy Iceberg validation commands
NikitaMatskevich Nov 9, 2026
a97456b
test: preserve native writes with disabled Bloom filters
NikitaMatskevich Nov 16, 2026
8f0ff78
fix: account for Iceberg Bloom filter memory
NikitaMatskevich Nov 16, 2026
ddf9b80
docs: explain Iceberg Bloom filter placement
NikitaMatskevich Nov 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions docs/source/contributor-guide/benchmarking.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,47 @@ These can also be run as a suite on a dedicated machine, see
[Micro Benchmarking on AWS EC2](benchmarking_micro_ec2.md). Published results are in
[benchmarks/results/micro](https://github.com/apache/datafusion-comet/tree/main/benchmarks/results/micro).

## Iceberg Bloom-filter writes

`CometIcebergWriteBenchmark` compares Spark and Comet Iceberg writers. Its `bloom` mode compares
native writes with Bloom filters off/on (requiring Iceberg 1.5+), using the same prepared Parquet source in eight files. Both cases enable native writes and the Bloom folding opt-in; only the table's Bloom
properties differ. Filters cover the bigint, string, and binary columns at the default 1 MiB cap,
without explicit NDV/FPP. The benchmark checks the native plan, row count, and filter presence
before measuring. Table creation and source preparation are outside the timed insert; the Iceberg
commit is included.

Use the existing benchmark target for timing, optionally followed by rows and minimum iterations:

```shell
make benchmark-org.apache.spark.sql.benchmark.CometIcebergWriteBenchmark bloom
```

Each case warms up and measures at least five writes of 4,194,304 rows by default. Use
`bloom-off` or `bloom-on` to select a single case, or append `32768 1` for a quick functional
check. Performance comparisons require a release build.

To collect peak RSS, build once and run each case in a separate JVM using the existing Maven
invocation. Obtain the shared JVM options from the Makefile:

```shell
make release PROFILES=-Pspark-4.1
export COMET_CONF_DIR="$PWD/conf"
export MAVEN_OPTS="$(make --no-print-directory print-benchmark-args PROFILES=-Pspark-4.1 BENCH_HEAP=8g | sed -n 's/^MAVEN_OPTS=//p')"
cd spark
/usr/bin/time -v ../mvnw exec:java -Dexec.classpathScope=test \
-Dexec.cleanupDaemonThreads=false \
-Dexec.mainClass=org.apache.spark.sql.benchmark.CometIcebergWriteBenchmark \
-Dexec.args="bloom-off" -Pspark-4.1
```

Repeat with `bloom-on`. On macOS, use `/usr/bin/time -l` instead of `-v`; its maximum RSS is in
bytes, while GNU time reports KiB. The reported RSS includes Maven, source generation, validation,
warmup, and writes; it is not the writer's allocation or its peak alone. Fresh processes avoid
carrying an earlier case's RSS high-water mark and retained native buffers into the next case.
The warehouse is local, so these results do not include object-store latency.

Do not measure RSS around `make benchmark-*`: its release-build prerequisite includes compilation.

```{toctree}
:hidden:

Expand Down
21 changes: 21 additions & 0 deletions docs/source/contributor-guide/iceberg-writes.md
Original file line number Diff line number Diff line change
Expand Up @@ -497,6 +497,27 @@ The Comet suites run against the Iceberg version each Spark profile pins in `spa
for Spark 3.5, 1.10.0 for 4.0, and 1.11.0 for 4.1. No Iceberg runtime is published
for Spark 4.2, so the 4.2 profile builds against the 4.0 runtime and the Iceberg suites skip there
(`icebergAvailable` is false on Spark 4.2). Only the default profile runs on every pull request.

Bloom-enabled native writes require an identifiable Iceberg version >= 1.5. See the
[user-guide runtime matrix](../user-guide/latest/iceberg.md#bloom-filters) for property and naming
rules. Ambiguous legacy aliases and invalid lookup names use Java. Iceberg 1.6/1.7 also uses Java
for explicit FPP properties because those releases validate them independently.

To exercise that independent FPP validation on Iceberg 1.7.2, first build Comet's JVM classes
and native library as described in the [Development Guide](development.md). Then run:

```shell
./mvnw -B test -Pspark-3.5 -Diceberg.spark.runtime.version=1.7.2 -Dtest=none \
'-Dsuites=org.apache.comet.serde.operator.IcebergWriteProtoTranslationSuite,org.apache.comet.CometIcebergWriteActionSuite legacy independent FPP validation,org.apache.comet.CometIcebergWriteDetectionSuite standalone Bloom shape properties'
```

The Spark 3.5 profile's `iceberg.spark.runtime.version` overrides only its test runtime; it
defaults to 1.8.1. This command runs the translation suite and the two integration tests for
independent FPP validation and standalone shape properties. Check that
`legacy independent FPP validation selects Java even without enabled columns` passes rather
than cancels: it cancels on the pinned runtime, which does not exercise the 1.6/1.7 code path.
Omit the override to return to the pinned version; no POM edits or Maven-cache cleanup are needed.

Behaviour that differs between Iceberg versions, such as reflection targets, metrics conventions
and partition path spelling, needs a test that runs on each. When you bump one of these pins, also
re-read how Iceberg's `ParquetWriter` and `Parquet.WriteBuilder` read their Hadoop `Configuration`,
Expand Down
Loading