Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .github/workflows/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ ruleset in `.asf.yaml`. That splits CI into three tiers:
- **Nightly tier** (`nightly`): the regression sweep of everything else, once
a day against `main` as it stands. The Comet test suites against the other
three Spark profiles, Spark SQL on Spark 3.5, 4.0 and 4.2, Iceberg
1.9/1.10, and the Delta contrib suites on Spark 3.5, 4.0 and 4.1. See
1.9/1.10/1.12, and the Delta contrib suites on Spark 3.5, 4.0 and 4.1. See
[Nightly tier](#nightly-tier) below for how a failure surfaces.

Every queue-only and nightly job has a `run-*` label that opts a pull request
Expand Down Expand Up @@ -119,6 +119,7 @@ tiers partition the list and that the `pr` tier is exactly the default profile.
spark_4_2 run-spark-4.2-tests
iceberg_1_9 run-iceberg-tests
iceberg_1_10 run-iceberg-tests
iceberg_1_12 run-iceberg-tests
delta_3_5 run-delta-tests
delta_4_0 run-delta-tests
delta_4_1 run-delta-tests
Expand Down Expand Up @@ -169,6 +170,7 @@ tiers partition the list and that the `pr` tier is exactly the default profile.
| `iceberg_1_11` | PR or merge group, paths matched | Iceberg sources |
| `iceberg_1_9` | nightly, **or** PR with `run-iceberg-tests` | Iceberg sources |
| `iceberg_1_10` | nightly, **or** PR with `run-iceberg-tests` | Iceberg sources |
| `iceberg_1_12` | nightly, **or** PR with `run-iceberg-tests` | Iceberg sources |
| `required_checks` | always, after every job above except `docs` | none (always runs) |
| `nightly_report` | schedule only, after `required_checks`, when it is not green | none |

Expand Down
18 changes: 18 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -299,6 +299,7 @@ jobs:
iceberg_1_9: ${{ steps.compute.outputs.iceberg_1_9 }}
iceberg_1_10: ${{ steps.compute.outputs.iceberg_1_10 }}
iceberg_1_11: ${{ steps.compute.outputs.iceberg_1_11 }}
iceberg_1_12: ${{ steps.compute.outputs.iceberg_1_12 }}
delta_3_5: ${{ steps.compute.outputs.delta_3_5 }}
delta_4_0: ${{ steps.compute.outputs.delta_4_0 }}
delta_4_1: ${{ steps.compute.outputs.delta_4_1 }}
Expand Down Expand Up @@ -619,6 +620,22 @@ jobs:
spark-full: '4.1.3'
java: 17

iceberg_1_12:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rationale is that no CI job covered 1.12, so nothing caught the silent fallback. Would this job have caught it? dev/ci/summarize-iceberg-writes.py never fails a job, and the upstream Iceberg tests do not assert which writer ran. I expect that reverting part 1 leaves iceberg_1_12 green, with the native share only visible in the step summary. The Comet suites that would fail still run on 1.11 only, since the spark-4.2 profile uses the 4.0 runtime. Could the job (or a Comet test) assert that at least one write ran natively, or that the executor reflection probe resolves against the 1.12 core jar? Otherwise the fix has no regression guard.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, it would not have caught it. Fixed in 6e991c5. summarize-iceberg-writes.py has a new --require-native flag that exits non-zero, after printing the summary, when none of the recorded writes ran on the native writer. That includes the case where no writes were recorded at all. The cross-shard summary in iceberg-spark-shard-coverage and the extensions job pass it, so the guard covers every Iceberg version, not just 1.12. Before relying on it, I ran the summarizer on last night's shard-1 reports. 1.9.1 had 316 native writes (97.5%) and 1.10.0 had 340 (97.7%). The 1.10.0 extensions run had 2927 (37.5%). So no version is near zero. A shard's own summary still never fails, because a single shard can legitimately record no writes. Unit tests for the flag are in test-summarize-iceberg-writes.py.

name: Iceberg Spark SQL Tests (Iceberg 1.12)
needs: changes
# Nightly by default; PRs need the `run-iceberg-tests` label. Iceberg 1.12
# is the first release with a Spark 4.2 module, and Spark 4.2 support is
# still experimental, so this sits in the nightly tier like the Spark 4.2
# SQL suite does.
if: needs.changes.outputs.iceberg_1_12 == 'true'
uses: ./.github/workflows/iceberg_spark_test_reusable.yml
with:
iceberg-short: '1.12'
iceberg-full: '1.12.0'
spark-short: '4.2'
spark-full: '4.2.0'
java: 17

# ---------------------------------------------------------------------------
# required_checks: the single context listed in `required_status_checks` for
# `main` in `.asf.yaml`, and therefore the only thing the merge queue waits
Expand Down Expand Up @@ -679,6 +696,7 @@ jobs:
- iceberg_1_9
- iceberg_1_10
- iceberg_1_11
- iceberg_1_12
runs-on: ubuntu-slim
steps:
- name: Summarize upstream results
Expand Down
7 changes: 5 additions & 2 deletions .github/workflows/iceberg_spark_test_reusable.yml
Original file line number Diff line number Diff line change
Expand Up @@ -200,8 +200,11 @@ jobs:
--task :iceberg-spark:iceberg-spark-${{ inputs.spark-short }}_${{ inputs.scala }}:test
- name: Summarize Iceberg writes across shards
if: ${{ !cancelled() }}
# --require-native fails the job when no write in any shard ran on the
# native writer. Iceberg's tests pass on either writer, so without it a
# release that makes the gate decline every write goes unnoticed.
run: |
python3 dev/ci/summarize-iceberg-writes.py --title "iceberg-spark, all shards" \
python3 dev/ci/summarize-iceberg-writes.py --require-native --title "iceberg-spark, all shards" \
iceberg-shard-reports

iceberg-spark-extensions:
Expand Down Expand Up @@ -242,7 +245,7 @@ jobs:
- name: Summarize Iceberg writes
if: ${{ !cancelled() }}
run: |
python3 dev/ci/summarize-iceberg-writes.py --title "iceberg-spark-extensions" \
python3 dev/ci/summarize-iceberg-writes.py --require-native --title "iceberg-spark-extensions" \
apache-iceberg/build/comet-iceberg-writes
- name: Upload Iceberg write report
if: ${{ !cancelled() }}
Expand Down
4 changes: 3 additions & 1 deletion dev/ci/check-ci-config.py
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,7 @@
"iceberg_1_9",
"iceberg_1_10",
"iceberg_1_11",
"iceberg_1_12",
}

# The two contrib/UDF gates also run ./mvnw, but consume no shared artifact.
Expand Down Expand Up @@ -230,14 +231,15 @@
"spark_4_2",
"iceberg_1_9",
"iceberg_1_10",
"iceberg_1_12",
"delta_3_5",
"delta_4_0",
"delta_4_1",
"build_linux_all_profiles",
}
# One label opts a pull request into every nightly Iceberg version. Iceberg
# 1.11 is in the PR tier, so it needs no label.
ICEBERG_OPT_IN = {"iceberg_1_9", "iceberg_1_10"}
ICEBERG_OPT_IN = {"iceberg_1_9", "iceberg_1_10", "iceberg_1_12"}
# Likewise one label for every Delta profile.
DELTA_OPT_IN = {"delta_3_5", "delta_4_0", "delta_4_1"}
ALL_JOBS = QUEUE_TIER | NIGHTLY_TIER | {"docs"}
Expand Down
65 changes: 13 additions & 52 deletions dev/ci/compute-changes.py
Original file line number Diff line number Diff line change
Expand Up @@ -335,56 +335,12 @@
".mvn/**",
"mvnw",
],
"iceberg_1_10": [
"native/**/src/**",
"native/**/Cargo.toml",
"native/Cargo.lock",
"common/src/main/**",
"common/pom.xml",
"spark/src/main/**",
"!spark/src/main/scala/org/apache/comet/GenerateDocs.scala",
"spark/pom.xml",
"dev/diffs/iceberg/**",
"pom.xml",
"rust-toolchain.toml",
".github/workflows/ci.yml",
".github/workflows/iceberg_spark_test_reusable.yml",
".github/actions/setup-builder/**",
".github/actions/setup-iceberg-builder/**",
"dev/ci/iceberg-test-shards.gradle",
"dev/ci/check-iceberg-shards.py",
"dev/ci/test-iceberg-shards.py",
".github/actions/upload-artifact-retry/**",
".github/actions/download-artifact-retry/**",
".github/actions/maven-bootstrap/**",
".mvn/**",
"mvnw",
],
"iceberg_1_11": [
"native/**/src/**",
"native/**/Cargo.toml",
"native/Cargo.lock",
"common/src/main/**",
"common/pom.xml",
"spark/src/main/**",
"!spark/src/main/scala/org/apache/comet/GenerateDocs.scala",
"spark/pom.xml",
"dev/diffs/iceberg/**",
"pom.xml",
"rust-toolchain.toml",
".github/workflows/ci.yml",
".github/workflows/iceberg_spark_test_reusable.yml",
".github/actions/setup-builder/**",
".github/actions/setup-iceberg-builder/**",
"dev/ci/iceberg-test-shards.gradle",
"dev/ci/check-iceberg-shards.py",
"dev/ci/test-iceberg-shards.py",
".github/actions/upload-artifact-retry/**",
".github/actions/download-artifact-retry/**",
".github/actions/maven-bootstrap/**",
".mvn/**",
"mvnw",
],
# The Iceberg versions share one path filter: each call builds the same
# Comet sources against a different Iceberg release. Populated below,
# after the dict, so the lists cannot drift.
"iceberg_1_10": [],
"iceberg_1_11": [],
"iceberg_1_12": [],
"delta_3_5": [
"contrib/delta-spark/**",
"!**.md",
Expand Down Expand Up @@ -417,6 +373,9 @@
FILTERS["build_linux_all_profiles"] = FILTERS["build_linux"]
FILTERS["delta_4_0"] = FILTERS["delta_3_5"]
FILTERS["delta_4_1"] = FILTERS["delta_3_5"]
FILTERS["iceberg_1_10"] = FILTERS["iceberg_1_9"]
FILTERS["iceberg_1_11"] = FILTERS["iceberg_1_9"]
FILTERS["iceberg_1_12"] = FILTERS["iceberg_1_9"]

# Which events may run each job, independent of the path filters above.
#
Expand Down Expand Up @@ -547,11 +506,13 @@
"spark_4_2": ["nightly", "label:run-spark-4.2-tests"],
# Same shape for Iceberg: 1.11 is the only Spark 4.1 coverage, so it is
# the one Iceberg version every pull request and the queue run, for the
# same reason as Spark 4.1 above. The two older versions run nightly,
# and one label brings both forward onto a pull request.
# same reason as Spark 4.1 above. The two older versions run nightly, as
# does 1.12, which is the Spark 4.2 coverage and so shares that suite's
# tier. One label brings all three forward onto a pull request.
"iceberg_1_9": ["nightly", "label:run-iceberg-tests"],
"iceberg_1_10": ["nightly", "label:run-iceberg-tests"],
"iceberg_1_11": ["pr", "queue"],
"iceberg_1_12": ["nightly", "label:run-iceberg-tests"],
# The Delta contrib suites all run nightly. The PR tier's strict warnings
# job compiles the contrib against Spark 3.5, so a change that breaks its
# build on Spark 3.5 fails on its own pull request. One label opts a pull
Expand Down
32 changes: 28 additions & 4 deletions dev/ci/summarize-iceberg-writes.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,13 @@
jobs does not count a shard twice. The latest attempt is the newest such
directory, whether or not it holds any report files, so a rerun that recorded
no writes is reported as missing rather than replaced by an earlier attempt.
The summary is also appended to $GITHUB_STEP_SUMMARY when that is set. It
never fails the job.
The summary is also appended to $GITHUB_STEP_SUMMARY when that is set.

By default it never fails the job. With --require-native it exits non-zero,
after printing the summary, when no write ran on Comet's native writer. That
guards against the native writer silently declining every write, as happened
when an Iceberg release removed a method the eligibility probe resolves: the
upstream tests still pass, because they do not assert which writer ran.
"""

import argparse
Expand All @@ -40,6 +45,7 @@
import os
from pathlib import Path
import re
import sys


SHARD_ATTEMPT = re.compile(r"-shard-(\d+)-attempt-(\d+)$")
Expand Down Expand Up @@ -153,19 +159,37 @@ def summarize(title, writes, missing=()):
return "\n".join(lines) + "\n"


def native_writes(writes):
return sum(1 for w in writes if w["writer"] == "native")


def main():
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
parser.add_argument("--title", required=True, help="heading for the summary")
parser.add_argument(
"--require-native",
action="store_true",
help="exit non-zero when no write ran on Comet's native writer",
)
parser.add_argument("roots", nargs="+", help="directories holding the report files")
args = parser.parse_args()

summary = summarize(args.title, *load(args.roots))
writes, missing = load(args.roots)
summary = summarize(args.title, writes, missing)
print(summary)
step_summary = os.environ.get("GITHUB_STEP_SUMMARY")
if step_summary:
with open(step_summary, "a", encoding="utf-8") as out:
out.write(summary + "\n")
if args.require_native and native_writes(writes) == 0:
print(
f"error: none of the {len(writes)} recorded Iceberg writes ran on Comet's native "
"writer, so the native writer is declining every write",
file=sys.stderr,
)
return 1
return 0


if __name__ == "__main__":
main()
sys.exit(main())
28 changes: 28 additions & 0 deletions dev/ci/test-summarize-iceberg-writes.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,12 @@

"""Fast regression tests for the Iceberg write report summary."""

import contextlib
import importlib.util
import io
import json
from pathlib import Path
import sys
import tempfile
import unittest

Expand Down Expand Up @@ -96,6 +99,31 @@ def test_files_outside_shard_artifacts_always_count(self):
(self.root / "iceberg-writes-local.jsonl").write_text(records("native", "jvm"))
self.assertEqual(self.load(), (["jvm", "native"], []))

def run_main(self, *args):
argv = sys.argv
sys.argv = ["summarize-iceberg-writes.py", "--title", "fixture", *args, str(self.root)]
try:
with contextlib.redirect_stdout(io.StringIO()), \
contextlib.redirect_stderr(io.StringIO()):
return SUMMARIZE.main()
finally:
sys.argv = argv

def test_require_native_passes_with_a_native_write(self):
self.attempt(1, 1, "jvm", "native")
self.attempt(2, 1, "jvm")
self.assertEqual(self.run_main("--require-native"), 0)

def test_require_native_fails_when_every_write_falls_back(self):
self.attempt(1, 1, "jvm", "spark")
self.attempt(2, 1, "jvm")
self.assertEqual(self.run_main("--require-native"), 1)
self.assertEqual(self.run_main(), 0)

def test_require_native_fails_when_no_writes_were_recorded(self):
self.attempt(1, 1)
self.assertEqual(self.run_main("--require-native"), 1)

def test_no_writes_at_all(self):
summary = SUMMARIZE.summarize("fixture", *SUMMARIZE.load([self.root / "missing"]))
self.assertIn("No Iceberg writes were recorded", summary)
Expand Down
Loading